Max level shown:
12025-02-28T12:00:00.540ZINFOnexus (BackgroundTasks): found new target blueprint (first find) background_task = blueprint_loader file = nexus/src/app/background/tasks/blueprint_load.rs:88 target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
22025-02-28T12:00:00.540ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 80.198536ms iteration = 6 status = {"enabled":false,"status":"first target blueprint","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z","time_found":"2025-02-28T12:00:00.540620583Z"}
32025-02-28T12:00:00.540ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 2 reason = Dependency
42025-02-28T12:00:00.540ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 2 reason = Dependency
52025-02-28T12:00:00.540ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
62025-02-28T12:00:00.540ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 9.27\u{b5}s iteration = 2 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
72025-02-28T12:00:00.540ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
82025-02-28T12:00:00.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
92025-02-28T12:00:00.541ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
102025-02-28T12:00:00.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
112025-02-28T12:00:00.541ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
122025-02-28T12:00:00.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
132025-02-28T12:00:00.541ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
142025-02-28T12:00:00.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
152025-02-28T12:00:00.541ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
162025-02-28T12:00:00.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
172025-02-28T12:00:00.543ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
182025-02-28T12:00:00.545ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
192025-02-28T12:00:00.545ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
202025-02-28T12:00:00.545ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
212025-02-28T12:00:00.545ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
222025-02-28T12:00:00.547ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
232025-02-28T12:00:00.547ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
242025-02-28T12:00:00.547ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
252025-02-28T12:00:00.547ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
262025-02-28T12:00:00.549ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
272025-02-28T12:00:00.549ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
282025-02-28T12:00:00.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
292025-02-28T12:00:00.549ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
302025-02-28T12:00:00.550ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
312025-02-28T12:00:00.550ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
322025-02-28T12:00:00.551ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
332025-02-28T12:00:00.551ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
342025-02-28T12:00:00.553ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
352025-02-28T12:00:00.553ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
362025-02-28T12:00:00.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
372025-02-28T12:00:00.597ZDEBGnexus (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[fd00:1122:3344:101::6]:32222/node/id
382025-02-28T12:00:00.636ZDEBGnexus (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[fd00:1122:3344:101::7]:32222/node/id
392025-02-28T12:00:00.675ZDEBGnexus (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[fd00:1122:3344:101::5]:32222/node/id
402025-02-28T12:00:00.716ZDEBGnexus (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[fd00:1122:3344:101::4]:32222/node/id
412025-02-28T12:00:00.757ZDEBGnexus (BackgroundTasks): client request background_task = crdb_node_id_collector body = None method = GET uri = http://[fd00:1122:3344:101::3]:32222/node/id
422025-02-28T12:00:01.552ZDEBGnexus (BackgroundTasks): client response background_task = crdb_node_id_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::7]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c5b40eb-79da-48de-b81f-4e2debac6157", "content-length": "64", "date": "Fri, 28 Feb 2025 12:00:00 GMT"} })
432025-02-28T12:00:01.552ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
442025-02-28T12:00:01.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
452025-02-28T12:00:01.555ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
462025-02-28T12:00:01.556ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
472025-02-28T12:00:01.556ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
482025-02-28T12:00:01.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
492025-02-28T12:00:01.666ZDEBGnexus (BackgroundTasks): client response background_task = crdb_node_id_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::5]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b4cb199-db0f-4dea-9b7e-70e82d0a7c4f", "content-length": "64", "date": "Fri, 28 Feb 2025 12:00:00 GMT"} })
502025-02-28T12:00:01.666ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
512025-02-28T12:00:01.666ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
522025-02-28T12:00:01.669ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
532025-02-28T12:00:01.671ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
542025-02-28T12:00:01.671ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
552025-02-28T12:00:01.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
562025-02-28T12:00:01.770ZDEBGnexus (BackgroundTasks): client response background_task = crdb_node_id_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::3]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "17e85c41-51e3-4770-bf92-6ae56e8de937", "content-length": "64", "date": "Fri, 28 Feb 2025 12:00:00 GMT"} })
572025-02-28T12:00:01.770ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
582025-02-28T12:00:01.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
592025-02-28T12:00:01.771ZDEBGnexus (BackgroundTasks): client response background_task = crdb_node_id_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::6]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5853e14-a5bf-4f2e-b07e-3d76738282a3", "content-length": "64", "date": "Fri, 28 Feb 2025 12:00:00 GMT"} })
602025-02-28T12:00:01.771ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
612025-02-28T12:00:01.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
622025-02-28T12:00:01.773ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
632025-02-28T12:00:01.775ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
642025-02-28T12:00:01.775ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
652025-02-28T12:00:01.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
662025-02-28T12:00:01.775ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
672025-02-28T12:00:01.777ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
682025-02-28T12:00:01.777ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
692025-02-28T12:00:01.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
702025-02-28T12:00:01.800ZDEBGnexus (BackgroundTasks): client response background_task = crdb_node_id_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::4]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cc2adf2-10b7-4273-9be1-dc9799da431d", "content-length": "64", "date": "Fri, 28 Feb 2025 12:00:00 GMT"} })
712025-02-28T12:00:01.800ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
722025-02-28T12:00:01.801ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
732025-02-28T12:00:01.802ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
742025-02-28T12:00:01.804ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
752025-02-28T12:00:01.804ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
762025-02-28T12:00:01.804ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
772025-02-28T12:00:01.811ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 1.270742866s iteration = 2 status = {"nsuccess":5}
782025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 2 reason = Timeout
792025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 3 reason = Timeout
802025-02-28T12:00:10.458ZWARNnexus (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:69 reason = no config
812025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 15.67\u{b5}s iteration = 3 status = {"error":"no config"}
822025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 3 reason = Timeout
832025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 3 reason = Timeout
842025-02-28T12:00:10.458ZWARNnexus (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:69 reason = no config
852025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 2.649\u{b5}s iteration = 3 status = {"error":"no config"}
862025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 3 reason = Timeout
872025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
882025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 2 reason = Timeout
892025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
902025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 2 reason = Timeout
912025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
922025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 2 reason = Timeout
932025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
942025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
952025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 2 reason = Timeout
962025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
972025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 2 reason = Timeout
982025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
992025-02-28T12:00:10.458ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-28 11:50:10.458176674 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
1002025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
1012025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1022025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1032025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1042025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
1052025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1062025-02-28T12:00:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
1072025-02-28T12:00:10.458ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5cefabf1-6831-46ff-98eb-1a3b5abad8b9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 845, tv_nsec: 91280456 } })
1082025-02-28T12:00:10.458ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5ebff2d9-48b6-4fe6-b852-684bb14b4955.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bddfb0f9-1d7b-45e4-b3a5-35aa6792956b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 845, tv_nsec: 91338677 } })
1092025-02-28T12:00:10.459ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
1102025-02-28T12:00:10.459ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 982.764\u{b5}s iteration = 2 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
1112025-02-28T12:00:10.459ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1122025-02-28T12:00:10.459ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
1132025-02-28T12:00:10.459ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.185899ms iteration = 2 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
1142025-02-28T12:00:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1152025-02-28T12:00:10.460ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1162025-02-28T12:00:10.460ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 7 reason = Timeout
1172025-02-28T12:00:10.460ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 3 reason = Timeout
1182025-02-28T12:00:10.460ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
1192025-02-28T12:00:10.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1202025-02-28T12:00:10.460ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 8.06\u{b5}s iteration = 3 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
1212025-02-28T12:00:10.460ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 3 reason = Timeout
1222025-02-28T12:00:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1232025-02-28T12:00:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1242025-02-28T12:00:10.462ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-28 11:50:10.458176674 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
1252025-02-28T12:00:10.462ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.395067ms iteration = 2 status = {"expiration":"2025-02-28T11:50:10.458176674Z","pruned":[]}
1262025-02-28T12:00:10.462ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1272025-02-28T12:00:10.462ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 2 reason = Timeout
1282025-02-28T12:00:10.462ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 4 reason = Timeout
1292025-02-28T12:00:10.462ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
1302025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1312025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 3 reason = Timeout
1322025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1332025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 3 reason = Timeout
1342025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 3 reason = Timeout
1352025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
1362025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1372025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 3 reason = Timeout
1382025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
1392025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 7 reason = Timeout
1402025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1412025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 3 reason = Timeout
1422025-02-28T12:00:10.463ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
1432025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1442025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 2 reason = Timeout
1452025-02-28T12:00:10.463ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
1462025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
1472025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 3 reason = Timeout
1482025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1492025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 3 reason = Timeout
1502025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1512025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 3 reason = Timeout
1522025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
1532025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1542025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 3 reason = Timeout
1552025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1562025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 3 reason = Timeout
1572025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 3 reason = Timeout
1582025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
1592025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
1602025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 3 reason = Timeout
1612025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
1622025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 2 reason = Timeout
1632025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
1642025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 3 reason = Timeout
1652025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
1662025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 2 reason = Timeout
1672025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
1682025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1692025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1702025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1712025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
1722025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
1732025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1742025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
1752025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1762025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1772025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1782025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1792025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
1802025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
1812025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1822025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
1832025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
1842025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1852025-02-28T12:00:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1862025-02-28T12:00:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
1872025-02-28T12:00:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
1882025-02-28T12:00:10.464ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1892025-02-28T12:00:10.465ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
1902025-02-28T12:00:10.465ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1912025-02-28T12:00:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1922025-02-28T12:00:10.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
1932025-02-28T12:00:10.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1942025-02-28T12:00:10.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
1952025-02-28T12:00:10.467ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1962025-02-28T12:00:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1972025-02-28T12:00:10.469ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1982025-02-28T12:00:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1992025-02-28T12:00:10.469ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
2002025-02-28T12:00:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2012025-02-28T12:00:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
2022025-02-28T12:00:10.471ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
2032025-02-28T12:00:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
2042025-02-28T12:00:10.472ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2052025-02-28T12:00:10.473ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 845, tv_nsec: 106143935 } })
2062025-02-28T12:00:10.473ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
2072025-02-28T12:00:10.474ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2082025-02-28T12:00:10.475ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
2092025-02-28T12:00:10.475ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
2102025-02-28T12:00:10.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
2112025-02-28T12:00:10.475ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 12.573803ms iteration = 2 status = {"errors":[],"found_port_ok":[]}
2122025-02-28T12:00:10.476ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
2132025-02-28T12:00:10.476ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2142025-02-28T12:00:10.476ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 13.432884ms iteration = 3 status = {"errors":[],"finish_invoked_ok":[]}
2152025-02-28T12:00:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2162025-02-28T12:00:10.478ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2172025-02-28T12:00:10.478ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
2182025-02-28T12:00:10.478ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
2192025-02-28T12:00:10.478ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2202025-02-28T12:00:10.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
2212025-02-28T12:00:10.479ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
2222025-02-28T12:00:10.479ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
2232025-02-28T12:00:10.479ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 16.007116ms iteration = 2 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
2242025-02-28T12:00:10.480ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 845, tv_nsec: 113016290 } })
2252025-02-28T12:00:10.480ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
2262025-02-28T12:00:10.480ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2272025-02-28T12:00:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
2282025-02-28T12:00:10.481ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2292025-02-28T12:00:10.481ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2302025-02-28T12:00:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2312025-02-28T12:00:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2322025-02-28T12:00:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2332025-02-28T12:00:10.481ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 18.60891ms iteration = 3 status = {"errors":[],"requests_created_ok":[]}
2342025-02-28T12:00:10.484ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
2352025-02-28T12:00:10.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
2362025-02-28T12:00:10.485ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 24.951182ms iteration = 3 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
2372025-02-28T12:00:10.486ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2382025-02-28T12:00:10.488ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2392025-02-28T12:00:10.489ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
2402025-02-28T12:00:10.489ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2412025-02-28T12:00:10.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2422025-02-28T12:00:10.492ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
2432025-02-28T12:00:10.492ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2442025-02-28T12:00:10.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2452025-02-28T12:00:10.493ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 845, tv_nsec: 125698677 } })
2462025-02-28T12:00:10.493ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
2472025-02-28T12:00:10.519ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2482025-02-28T12:00:10.519ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2492025-02-28T12:00:10.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2502025-02-28T12:00:10.520ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
2512025-02-28T12:00:10.520ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
2522025-02-28T12:00:10.520ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
2532025-02-28T12:00:10.520ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
2542025-02-28T12:00:10.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
2552025-02-28T12:00:10.521ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2562025-02-28T12:00:10.521ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
2572025-02-28T12:00:10.521ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
2582025-02-28T12:00:10.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
2592025-02-28T12:00:10.523ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
2602025-02-28T12:00:10.523ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
2612025-02-28T12:00:10.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
2622025-02-28T12:00:10.526ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2632025-02-28T12:00:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
2642025-02-28T12:00:10.528ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2652025-02-28T12:00:10.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2662025-02-28T12:00:10.533ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2672025-02-28T12:00:10.533ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
2682025-02-28T12:00:10.533ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
2692025-02-28T12:00:10.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
2702025-02-28T12:00:10.536ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 73.629177ms iteration = 3 status = {"errors":[],"garbage_collect_requested":[]}
2712025-02-28T12:00:10.537ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
2722025-02-28T12:00:10.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
2732025-02-28T12:00:10.539ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
2742025-02-28T12:00:10.539ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2752025-02-28T12:00:10.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
2762025-02-28T12:00:10.542ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 79.210252ms iteration = 3 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
2772025-02-28T12:00:10.542ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
2782025-02-28T12:00:10.542ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2792025-02-28T12:00:10.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2802025-02-28T12:00:10.543ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
2812025-02-28T12:00:10.543ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
2822025-02-28T12:00:10.543ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 79.998361ms iteration = 2 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
2832025-02-28T12:00:10.544ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2842025-02-28T12:00:10.545ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
2852025-02-28T12:00:10.545ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
2862025-02-28T12:00:10.545ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
2872025-02-28T12:00:10.558ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
2882025-02-28T12:00:10.558ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
2892025-02-28T12:00:10.558ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c9735b9-d9f2-4be5-8da5-25b83be12d07", "content-length": "26", "date": "Fri, 28 Feb 2025 12:00:10 GMT"} })
2902025-02-28T12:00:10.558ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
2912025-02-28T12:00:10.560ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
2922025-02-28T12:00:10.560ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
2932025-02-28T12:00:10.560ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
2942025-02-28T12:00:10.564ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
2952025-02-28T12:00:10.564ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
2962025-02-28T12:00:10.564ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a46a5893-ff19-400a-8882-b93cd7af1c2c", "content-length": "26", "date": "Fri, 28 Feb 2025 12:00:10 GMT"} })
2972025-02-28T12:00:10.564ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
2982025-02-28T12:00:10.566ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
2992025-02-28T12:00:10.574ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
3002025-02-28T12:00:10.574ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
3012025-02-28T12:00:10.574ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
3022025-02-28T12:00:10.574ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3032025-02-28T12:00:10.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3042025-02-28T12:00:10.611ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
3052025-02-28T12:00:10.611ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
3062025-02-28T12:00:10.611ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "49be4f4b-c7ff-43ac-9529-26817ba37d2e", "content-length": "26", "date": "Fri, 28 Feb 2025 12:00:10 GMT"} })
3072025-02-28T12:00:10.611ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
3082025-02-28T12:00:10.613ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
3092025-02-28T12:00:10.619ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
3102025-02-28T12:00:10.619ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
3112025-02-28T12:00:10.620ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "cacb3389-6ba2-4bec-a66b-f3875a0524c8", "content-length": "372", "date": "Fri, 28 Feb 2025 12:00:10 GMT"} })
3122025-02-28T12:00:10.623ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3132025-02-28T12:00:10.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3142025-02-28T12:00:10.624ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 161.500498ms iteration = 7 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
3152025-02-28T12:00:10.629ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3162025-02-28T12:00:10.636ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
3172025-02-28T12:00:10.636ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3182025-02-28T12:00:10.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3192025-02-28T12:00:10.639ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3202025-02-28T12:00:10.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3212025-02-28T12:00:10.649ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
3222025-02-28T12:00:10.649ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 186.212704ms iteration = 3 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
3232025-02-28T12:00:10.650ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3242025-02-28T12:00:10.655ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3252025-02-28T12:00:10.655ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3262025-02-28T12:00:10.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3272025-02-28T12:00:10.686ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
3282025-02-28T12:00:10.687ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b00f522-f5d4-49dd-88cb-66acef6837c7", "content-length": "1", "date": "Fri, 28 Feb 2025 12:00:09 GMT"} })
3292025-02-28T12:00:10.692ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3302025-02-28T12:00:10.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3312025-02-28T12:00:10.694ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3322025-02-28T12:00:10.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3332025-02-28T12:00:10.704ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
3342025-02-28T12:00:10.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
3352025-02-28T12:00:10.705ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3362025-02-28T12:00:10.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3372025-02-28T12:00:10.706ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3382025-02-28T12:00:10.709ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 251.825359ms iteration = 3 status = {}
3392025-02-28T12:00:10.711ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
3402025-02-28T12:00:10.711ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
3412025-02-28T12:00:10.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
3422025-02-28T12:00:10.724ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
3432025-02-28T12:00:10.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
3442025-02-28T12:00:10.725ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 262.291672ms iteration = 3 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
3452025-02-28T12:00:10.725ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
3462025-02-28T12:00:10.725ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
3472025-02-28T12:00:10.726ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3482025-02-28T12:00:10.727ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3492025-02-28T12:00:10.729ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3502025-02-28T12:00:10.729ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 266.839932ms iteration = 4 status = {"physical_disks_added":0}
3512025-02-28T12:00:10.736ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
3522025-02-28T12:00:10.736ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3532025-02-28T12:00:10.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3542025-02-28T12:00:10.746ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
3552025-02-28T12:00:10.746ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 0 version = 2
3562025-02-28T12:00:10.746ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3572025-02-28T12:00:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3582025-02-28T12:00:10.759ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3592025-02-28T12:00:10.761ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
3602025-02-28T12:00:10.761ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
3612025-02-28T12:00:10.761ZINFOnexus (BackgroundTasks): found latest generation (first find) background_task = dns_config_internal dns_group = internal file = nexus/src/app/background/tasks/dns_config.rs:87 generation = 1
3622025-02-28T12:00:10.761ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 303.761533ms iteration = 2 status = {"generation":1}
3632025-02-28T12:00:10.761ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 4 reason = Dependency
3642025-02-28T12:00:10.763ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3652025-02-28T12:00:10.763ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3662025-02-28T12:00:10.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3672025-02-28T12:00:10.771ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 1 version = 2
3682025-02-28T12:00:10.771ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
3692025-02-28T12:00:10.771ZINFOnexus (BackgroundTasks): found latest generation (first find) background_task = dns_config_external dns_group = external file = nexus/src/app/background/tasks/dns_config.rs:87 generation = 2
3702025-02-28T12:00:10.771ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 313.739613ms iteration = 2 status = {"generation":2}
3712025-02-28T12:00:10.771ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 4 reason = Dependency
3722025-02-28T12:00:10.775ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
3732025-02-28T12:00:10.775ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
3742025-02-28T12:00:10.775ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 312.240237ms iteration = 3 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
3752025-02-28T12:00:10.781ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3762025-02-28T12:00:10.784ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
3772025-02-28T12:00:10.784ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3782025-02-28T12:00:10.784ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
3792025-02-28T12:00:10.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3802025-02-28T12:00:10.784ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 321.405329ms iteration = 3 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
3812025-02-28T12:00:10.787ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 324.640836ms iteration = 3 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
3822025-02-28T12:00:10.805ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 347.57345ms iteration = 3 status = null
3832025-02-28T12:00:10.815ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
3842025-02-28T12:00:10.815ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 355.215025ms iteration = 7 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
3852025-02-28T12:00:10.847ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
3862025-02-28T12:00:10.852ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
3872025-02-28T12:00:10.852ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 389.071912ms iteration = 2 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
3882025-02-28T12:00:10.856ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
3892025-02-28T12:00:10.856ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
3902025-02-28T12:00:10.856ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "5e3417c0-3372-4752-a192-ec4f8b01ef67", "date": "Fri, 28 Feb 2025 12:00:10 GMT"} })
3912025-02-28T12:00:10.856ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3922025-02-28T12:00:10.857ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3932025-02-28T12:00:10.860ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3942025-02-28T12:00:10.860ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3952025-02-28T12:00:10.863ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3962025-02-28T12:00:10.864ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3972025-02-28T12:00:10.868ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
3982025-02-28T12:00:10.868ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
3992025-02-28T12:00:10.868ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "aa9c80a7-b283-4aa9-8ea2-63c22ff1799d", "date": "Fri, 28 Feb 2025 12:00:10 GMT"} })
4002025-02-28T12:00:10.870ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 407.168909ms iteration = 3 status = {}
4012025-02-28T12:00:10.874ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (initial) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-28T11:59:08.719399Z, time_modified: 2025-02-28T11:59:08.719399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:77
4022025-02-28T12:00:10.874ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 416.467383ms iteration = 2 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]}},"default_endpoint":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
4032025-02-28T12:00:10.884ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
4042025-02-28T12:00:10.896ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4052025-02-28T12:00:10.897ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4062025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = / versions = all
4072025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /access versions = all
4082025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /assets/{path} versions = all
4092025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /device/auth versions = all
4102025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /device/confirm versions = all
4112025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /device/success versions = all
4122025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /device/token versions = all
4132025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /device/verify versions = all
4142025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /experimental/v1/probes versions = all
4152025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /experimental/v1/probes versions = all
4162025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /experimental/v1/probes/{probe} versions = all
4172025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /experimental/v1/probes/{probe} versions = all
4182025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /experimental/v1/system/support-bundles versions = all
4192025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /experimental/v1/system/support-bundles versions = all
4202025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
4212025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
4222025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
4232025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
4242025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
4252025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
4262025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
4272025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /images versions = all
4282025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /login versions = all
4292025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /login/{silo_name}/local versions = all
4302025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
4312025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
4322025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
4332025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /lookup/{path} versions = all
4342025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /projects/{path} versions = all
4352025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /projects-new versions = all
4362025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /settings/{path} versions = all
4372025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /system/{path} versions = all
4382025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /utilization versions = all
4392025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/affinity-groups versions = all
4402025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/affinity-groups versions = all
4412025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
4422025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
4432025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
4442025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
4452025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
4462025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
4472025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
4482025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/anti-affinity-groups versions = all
4492025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/anti-affinity-groups versions = all
4502025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
4512025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
4522025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
4532025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
4542025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
4552025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
4562025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
4572025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/certificates versions = all
4582025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/certificates versions = all
4592025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/certificates/{certificate} versions = all
4602025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/certificates/{certificate} versions = all
4612025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/disks versions = all
4622025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/disks versions = all
4632025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/disks/{disk} versions = all
4642025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/disks/{disk} versions = all
4652025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/disks/{disk}/bulk-write versions = all
4662025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
4672025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
4682025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/disks/{disk}/finalize versions = all
4692025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
4702025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/floating-ips versions = all
4712025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/floating-ips versions = all
4722025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
4732025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/floating-ips/{floating_ip} versions = all
4742025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
4752025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
4762025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
4772025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/groups versions = all
4782025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/groups/{group_id} versions = all
4792025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/images versions = all
4802025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/images versions = all
4812025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/images/{image} versions = all
4822025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/images/{image} versions = all
4832025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/images/{image}/demote versions = all
4842025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/images/{image}/promote versions = all
4852025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/instances versions = all
4862025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/instances versions = all
4872025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/instances/{instance} versions = all
4882025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/instances/{instance} versions = all
4892025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/instances/{instance} versions = all
4902025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/instances/{instance}/disks versions = all
4912025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/instances/{instance}/disks/attach versions = all
4922025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/instances/{instance}/disks/detach versions = all
4932025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/instances/{instance}/external-ips versions = all
4942025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
4952025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
4962025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/instances/{instance}/reboot versions = all
4972025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/instances/{instance}/serial-console versions = all
4982025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
4992025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
5002025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/instances/{instance}/start versions = all
5012025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/instances/{instance}/stop versions = all
5022025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/internet-gateway-ip-addresses versions = all
5032025-02-28T12:00:10.900ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5042025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/internet-gateway-ip-addresses versions = all
5052025-02-28T12:00:10.900ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
5062025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/internet-gateway-ip-pools versions = all
5072025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/internet-gateway-ip-pools versions = all
5082025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
5092025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/internet-gateways versions = all
5102025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/internet-gateways versions = all
5112025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
5122025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/internet-gateways/{gateway} versions = all
5132025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/ip-pools versions = all
5142025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/ip-pools/{pool} versions = all
5152025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/login/{silo_name}/local versions = all
5162025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/logout versions = all
5172025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/me versions = all
5182025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/me/groups versions = all
5192025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/me/ssh-keys versions = all
5202025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/me/ssh-keys versions = all
5212025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
5222025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
5232025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/metrics/{metric_name} versions = all
5242025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/network-interfaces versions = all
5252025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/network-interfaces versions = all
5262025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/network-interfaces/{interface} versions = all
5272025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/network-interfaces/{interface} versions = all
5282025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/network-interfaces/{interface} versions = all
5292025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/ping versions = all
5302025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/policy versions = all
5312025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/policy versions = all
5322025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/projects versions = all
5332025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/projects versions = all
5342025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/projects/{project} versions = all
5352025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/projects/{project} versions = all
5362025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/projects/{project} versions = all
5372025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/projects/{project}/policy versions = all
5382025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/projects/{project}/policy versions = all
5392025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/snapshots versions = all
5402025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/snapshots versions = all
5412025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/snapshots/{snapshot} versions = all
5422025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/snapshots/{snapshot} versions = all
5432025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/hardware/disks versions = all
5442025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
5452025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
5462025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/hardware/racks versions = all
5472025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
5482025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/hardware/sleds versions = all
5492025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/hardware/sleds versions = all
5502025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
5512025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
5522025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
5532025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
5542025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
5552025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/hardware/switch-port versions = all
5562025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
5572025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
5582025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
5592025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
5602025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
5612025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/hardware/switches versions = all
5622025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
5632025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/identity-providers versions = all
5642025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/identity-providers/local/users versions = all
5652025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
5662025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
5672025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/identity-providers/saml versions = all
5682025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
5692025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/ip-pools versions = all
5702025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/ip-pools versions = all
5712025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
5722025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/ip-pools/{pool} versions = all
5732025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/system/ip-pools/{pool} versions = all
5742025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
5752025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
5762025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
5772025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
5782025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
5792025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
5802025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
5812025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
5822025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/ip-pools-service versions = all
5832025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/ip-pools-service/ranges versions = all
5842025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
5852025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
5862025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/metrics/{metric_name} versions = all
5872025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/networking/address-lot versions = all
5882025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/networking/address-lot versions = all
5892025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
5902025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
5912025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/networking/allow-list versions = all
5922025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/system/networking/allow-list versions = all
5932025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/networking/bfd-disable versions = all
5942025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/networking/bfd-enable versions = all
5952025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/networking/bfd-status versions = all
5962025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/system/networking/bgp versions = all
5972025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/networking/bgp versions = all
5982025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/networking/bgp versions = all
5992025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/networking/bgp-announce-set versions = all
6002025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
6012025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
6022025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
6032025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/networking/bgp-exported versions = all
6042025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/networking/bgp-message-history versions = all
6052025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
6062025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/networking/bgp-status versions = all
6072025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/networking/loopback-address versions = all
6082025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/networking/loopback-address versions = all
6092025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
6102025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
6112025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/networking/switch-port-settings versions = all
6122025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/networking/switch-port-settings versions = all
6132025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
6142025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/policy versions = all
6152025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/system/policy versions = all
6162025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/roles versions = all
6172025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/roles/{role_name} versions = all
6182025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/silo-quotas versions = all
6192025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/silos versions = all
6202025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/silos versions = all
6212025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/system/silos/{silo} versions = all
6222025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/silos/{silo} versions = all
6232025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
6242025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/silos/{silo}/policy versions = all
6252025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/system/silos/{silo}/policy versions = all
6262025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/silos/{silo}/quotas versions = all
6272025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
6282025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/system/timeseries/query versions = all
6292025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/timeseries/schemas versions = all
6302025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/system/update/repository versions = all
6312025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/update/repository/{system_version} versions = all
6322025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/users versions = all
6332025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/users/{user_id} versions = all
6342025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/users-builtin versions = all
6352025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/users-builtin/{user} versions = all
6362025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/utilization/silos versions = all
6372025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/system/utilization/silos/{silo} versions = all
6382025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/timeseries/query versions = all
6392025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/users versions = all
6402025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/utilization versions = all
6412025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/vpc-firewall-rules versions = all
6422025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/vpc-firewall-rules versions = all
6432025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/vpc-router-routes versions = all
6442025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/vpc-router-routes versions = all
6452025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
6462025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/vpc-router-routes/{route} versions = all
6472025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/vpc-router-routes/{route} versions = all
6482025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/vpc-routers versions = all
6492025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/vpc-routers versions = all
6502025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/vpc-routers/{router} versions = all
6512025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/vpc-routers/{router} versions = all
6522025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/vpc-routers/{router} versions = all
6532025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/vpc-subnets versions = all
6542025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/vpc-subnets versions = all
6552025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
6562025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/vpc-subnets/{subnet} versions = all
6572025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
6582025-02-28T12:00:10.901ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
6592025-02-28T12:00:10.902ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/vpcs versions = all
6602025-02-28T12:00:10.902ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = POST path = /v1/vpcs versions = all
6612025-02-28T12:00:10.902ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = DELETE path = /v1/vpcs/{vpc} versions = all
6622025-02-28T12:00:10.902ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = GET path = /v1/vpcs/{vpc} versions = all
6632025-02-28T12:00:10.902ZDEBGnexus (dropshot_external): registered endpoint local_addr = 172.30.2.5:443 method = PUT path = /v1/vpcs/{vpc} versions = all
6642025-02-28T12:00:10.902ZINFOnexus (dropshot_external): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = 172.30.2.5:443
6652025-02-28T12:00:10.902ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6662025-02-28T12:00:10.902ZDEBGnexus (dropshot_external): successfully registered DTrace USDT probes local_addr = 172.30.2.5:443
6672025-02-28T12:00:10.902ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6682025-02-28T12:00:10.905ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
6692025-02-28T12:00:10.906ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6702025-02-28T12:00:10.906ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6712025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = / versions = all
6722025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /access versions = all
6732025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /assets/{path} versions = all
6742025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /device/auth versions = all
6752025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /device/confirm versions = all
6762025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /device/success versions = all
6772025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /device/token versions = all
6782025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /device/verify versions = all
6792025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /experimental/v1/probes versions = all
6802025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /experimental/v1/probes versions = all
6812025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /experimental/v1/probes/{probe} versions = all
6822025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /experimental/v1/probes/{probe} versions = all
6832025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /experimental/v1/system/support-bundles versions = all
6842025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /experimental/v1/system/support-bundles versions = all
6852025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
6862025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
6872025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
6882025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
6892025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
6902025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
6912025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
6922025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /images versions = all
6932025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /login versions = all
6942025-02-28T12:00:10.910ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /login/{silo_name}/local versions = all
6952025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
6962025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
6972025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
6982025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /lookup/{path} versions = all
6992025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /projects/{path} versions = all
7002025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /projects-new versions = all
7012025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /settings/{path} versions = all
7022025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /system/{path} versions = all
7032025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /utilization versions = all
7042025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/affinity-groups versions = all
7052025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/affinity-groups versions = all
7062025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
7072025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
7082025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
7092025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
7102025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
7112025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
7122025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
7132025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/anti-affinity-groups versions = all
7142025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/anti-affinity-groups versions = all
7152025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
7162025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
7172025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
7182025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
7192025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
7202025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
7212025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
7222025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/certificates versions = all
7232025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/certificates versions = all
7242025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/certificates/{certificate} versions = all
7252025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/certificates/{certificate} versions = all
7262025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/disks versions = all
7272025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/disks versions = all
7282025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/disks/{disk} versions = all
7292025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/disks/{disk} versions = all
7302025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/disks/{disk}/bulk-write versions = all
7312025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
7322025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
7332025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/disks/{disk}/finalize versions = all
7342025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
7352025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/floating-ips versions = all
7362025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/floating-ips versions = all
7372025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
7382025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/floating-ips/{floating_ip} versions = all
7392025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
7402025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
7412025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
7422025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/groups versions = all
7432025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/groups/{group_id} versions = all
7442025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/images versions = all
7452025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/images versions = all
7462025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/images/{image} versions = all
7472025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/images/{image} versions = all
7482025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/images/{image}/demote versions = all
7492025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/images/{image}/promote versions = all
7502025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/instances versions = all
7512025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/instances versions = all
7522025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/instances/{instance} versions = all
7532025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/instances/{instance} versions = all
7542025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/instances/{instance} versions = all
7552025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/instances/{instance}/disks versions = all
7562025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/instances/{instance}/disks/attach versions = all
7572025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/instances/{instance}/disks/detach versions = all
7582025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/instances/{instance}/external-ips versions = all
7592025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
7602025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
7612025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/instances/{instance}/reboot versions = all
7622025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/instances/{instance}/serial-console versions = all
7632025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
7642025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
7652025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/instances/{instance}/start versions = all
7662025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/instances/{instance}/stop versions = all
7672025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/internet-gateway-ip-addresses versions = all
7682025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/internet-gateway-ip-addresses versions = all
7692025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
7702025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/internet-gateway-ip-pools versions = all
7712025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/internet-gateway-ip-pools versions = all
7722025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
7732025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/internet-gateways versions = all
7742025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/internet-gateways versions = all
7752025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
7762025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/internet-gateways/{gateway} versions = all
7772025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/ip-pools versions = all
7782025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/ip-pools/{pool} versions = all
7792025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/login/{silo_name}/local versions = all
7802025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/logout versions = all
7812025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/me versions = all
7822025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/me/groups versions = all
7832025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/me/ssh-keys versions = all
7842025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/me/ssh-keys versions = all
7852025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
7862025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
7872025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/metrics/{metric_name} versions = all
7882025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/network-interfaces versions = all
7892025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/network-interfaces versions = all
7902025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/network-interfaces/{interface} versions = all
7912025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/network-interfaces/{interface} versions = all
7922025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/network-interfaces/{interface} versions = all
7932025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/ping versions = all
7942025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/policy versions = all
7952025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/policy versions = all
7962025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/projects versions = all
7972025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/projects versions = all
7982025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/projects/{project} versions = all
7992025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/projects/{project} versions = all
8002025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/projects/{project} versions = all
8012025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/projects/{project}/policy versions = all
8022025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/projects/{project}/policy versions = all
8032025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/snapshots versions = all
8042025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/snapshots versions = all
8052025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/snapshots/{snapshot} versions = all
8062025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/snapshots/{snapshot} versions = all
8072025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/hardware/disks versions = all
8082025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
8092025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
8102025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/hardware/racks versions = all
8112025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
8122025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/hardware/sleds versions = all
8132025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/hardware/sleds versions = all
8142025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
8152025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
8162025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
8172025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
8182025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
8192025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/hardware/switch-port versions = all
8202025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
8212025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
8222025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
8232025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
8242025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
8252025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/hardware/switches versions = all
8262025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
8272025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/identity-providers versions = all
8282025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/identity-providers/local/users versions = all
8292025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
8302025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
8312025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/identity-providers/saml versions = all
8322025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
8332025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/ip-pools versions = all
8342025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/ip-pools versions = all
8352025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
8362025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/ip-pools/{pool} versions = all
8372025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/system/ip-pools/{pool} versions = all
8382025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
8392025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
8402025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
8412025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
8422025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
8432025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
8442025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
8452025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
8462025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/ip-pools-service versions = all
8472025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/ip-pools-service/ranges versions = all
8482025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
8492025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
8502025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/metrics/{metric_name} versions = all
8512025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/networking/address-lot versions = all
8522025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/networking/address-lot versions = all
8532025-02-28T12:00:10.911ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
8542025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
8552025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/networking/allow-list versions = all
8562025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/system/networking/allow-list versions = all
8572025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/networking/bfd-disable versions = all
8582025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/networking/bfd-enable versions = all
8592025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/networking/bfd-status versions = all
8602025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/system/networking/bgp versions = all
8612025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/networking/bgp versions = all
8622025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/networking/bgp versions = all
8632025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/networking/bgp-announce-set versions = all
8642025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
8652025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
8662025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
8672025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/networking/bgp-exported versions = all
8682025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/networking/bgp-message-history versions = all
8692025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
8702025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/networking/bgp-status versions = all
8712025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/networking/loopback-address versions = all
8722025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/networking/loopback-address versions = all
8732025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
8742025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
8752025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/networking/switch-port-settings versions = all
8762025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/networking/switch-port-settings versions = all
8772025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
8782025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/policy versions = all
8792025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/system/policy versions = all
8802025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/roles versions = all
8812025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/roles/{role_name} versions = all
8822025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/silo-quotas versions = all
8832025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/silos versions = all
8842025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/silos versions = all
8852025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/system/silos/{silo} versions = all
8862025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/silos/{silo} versions = all
8872025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
8882025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/silos/{silo}/policy versions = all
8892025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/system/silos/{silo}/policy versions = all
8902025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/silos/{silo}/quotas versions = all
8912025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
8922025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/system/timeseries/query versions = all
8932025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/timeseries/schemas versions = all
8942025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/system/update/repository versions = all
8952025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/update/repository/{system_version} versions = all
8962025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/users versions = all
8972025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/users/{user_id} versions = all
8982025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/users-builtin versions = all
8992025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/users-builtin/{user} versions = all
9002025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/utilization/silos versions = all
9012025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/system/utilization/silos/{silo} versions = all
9022025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/timeseries/query versions = all
9032025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/users versions = all
9042025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/utilization versions = all
9052025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/vpc-firewall-rules versions = all
9062025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/vpc-firewall-rules versions = all
9072025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/vpc-router-routes versions = all
9082025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/vpc-router-routes versions = all
9092025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
9102025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/vpc-router-routes/{route} versions = all
9112025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/vpc-router-routes/{route} versions = all
9122025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/vpc-routers versions = all
9132025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/vpc-routers versions = all
9142025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/vpc-routers/{router} versions = all
9152025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/vpc-routers/{router} versions = all
9162025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/vpc-routers/{router} versions = all
9172025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/vpc-subnets versions = all
9182025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/vpc-subnets versions = all
9192025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
9202025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/vpc-subnets/{subnet} versions = all
9212025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
9222025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
9232025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/vpcs versions = all
9242025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = POST path = /v1/vpcs versions = all
9252025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = DELETE path = /v1/vpcs/{vpc} versions = all
9262025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = GET path = /v1/vpcs/{vpc} versions = all
9272025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): registered endpoint local_addr = [fd00:1122:3344:101::a]:12228 method = PUT path = /v1/vpcs/{vpc} versions = all
9282025-02-28T12:00:10.912ZINFOnexus (dropshot_external_techport): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [fd00:1122:3344:101::a]:12228
9292025-02-28T12:00:10.912ZDEBGnexus (dropshot_external_techport): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::a]:12228
9302025-02-28T12:00:10.914ZDEBGnexus (nexus-producer-server): registered DTrace probes
9312025-02-28T12:00:10.919ZDEBGnexus (dropshot): registered endpoint local_addr = [fd00:1122:3344:101::a]:56450 method = GET path = /{producer_id} versions = all
9322025-02-28T12:00:10.919ZINFOnexus (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [fd00:1122:3344:101::a]:56450
9332025-02-28T12:00:10.920ZDEBGnexus (dropshot): successfully registered DTrace USDT probes local_addr = [fd00:1122:3344:101::a]:56450
9342025-02-28T12:00:10.920ZDEBGnexus (nexus-producer-server): Requested any available port, Dropshot server has been bound to [fd00:1122:3344:101::a]:56450
9352025-02-28T12:00:10.920ZDEBGnexus (nexus-producer-server): Nexus IP provided explicitly, will use it addr = [fd00:1122:3344:101::a]:12221
9362025-02-28T12:00:10.920ZDEBGnexus (nexus-producer-server): starting producer registration task
9372025-02-28T12:00:10.921ZINFOnexus (nexus-producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::a]:56450 file = oximeter/producer/src/lib.rs:279 interval = 10s producer_id = c1b34bc5-1c2f-4d3c-808c-b7a441aac295
9382025-02-28T12:00:10.921ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
9392025-02-28T12:00:10.921ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::a]:12221
9402025-02-28T12:00:10.955ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
9412025-02-28T12:00:10.979ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9422025-02-28T12:00:10.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9432025-02-28T12:00:10.981ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9442025-02-28T12:00:10.983ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
9452025-02-28T12:00:10.983ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9462025-02-28T12:00:10.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9472025-02-28T12:00:10.988ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9482025-02-28T12:00:10.990ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
9492025-02-28T12:00:10.990ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9502025-02-28T12:00:10.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9512025-02-28T12:00:11.006ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
9522025-02-28T12:00:11.007ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "4b6e7a83-2083-4e0a-a3bc-f6bd642c1b35", "date": "Fri, 28 Feb 2025 12:00:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
9532025-02-28T12:00:11.007ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
9542025-02-28T12:00:11.008ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "c45de19d-73da-47a7-bc59-f22424b560f5", "date": "Fri, 28 Feb 2025 12:00:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
9552025-02-28T12:00:11.008ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
9562025-02-28T12:00:11.009ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9572025-02-28T12:00:11.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9582025-02-28T12:00:11.010ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
9592025-02-28T12:00:11.010ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 238.632771ms iteration = 4 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
9602025-02-28T12:00:11.016ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9612025-02-28T12:00:11.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9622025-02-28T12:00:11.016ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9632025-02-28T12:00:11.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9642025-02-28T12:00:11.019ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
9652025-02-28T12:00:11.019ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9662025-02-28T12:00:11.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9672025-02-28T12:00:11.030ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9682025-02-28T12:00:11.030ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9692025-02-28T12:00:11.034ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
9702025-02-28T12:00:11.034ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::a]:59832
9712025-02-28T12:00:11.037ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
9722025-02-28T12:00:11.039ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9732025-02-28T12:00:11.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9742025-02-28T12:00:11.040ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:59832 req_id = 1be80351-2ec9-4997-a08d-bc72b8c984b2 roles = RoleSet { roles: {} } uri = /metrics/producers
9752025-02-28T12:00:11.040ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:59832 req_id = 1be80351-2ec9-4997-a08d-bc72b8c984b2 resource = Database result = Ok(()) uri = /metrics/producers
9762025-02-28T12:00:11.042ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "50605b74-f47f-4bcf-868f-2050cb0441cf", "content-length": "134", "date": "Fri, 28 Feb 2025 12:00:10 GMT"} })
9772025-02-28T12:00:11.042ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
9782025-02-28T12:00:11.042ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
9792025-02-28T12:00:11.042ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
9802025-02-28T12:00:11.043ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "3e9c4dba-dcc1-4d9b-b170-8429edace6a1", "date": "Fri, 28 Feb 2025 12:00:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
9812025-02-28T12:00:11.043ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
9822025-02-28T12:00:11.044ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "27d70cea-c18d-42f4-a2d9-55f2fa98874e", "date": "Fri, 28 Feb 2025 12:00:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
9832025-02-28T12:00:11.044ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
9842025-02-28T12:00:11.044ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
9852025-02-28T12:00:11.044ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
9862025-02-28T12:00:11.044ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9872025-02-28T12:00:11.044ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9882025-02-28T12:00:11.044ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9892025-02-28T12:00:11.044ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
9902025-02-28T12:00:11.045ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "f8843bec-db13-4372-8c16-0075f8906fe5", "date": "Fri, 28 Feb 2025 12:00:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
9912025-02-28T12:00:11.045ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
9922025-02-28T12:00:11.047ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9932025-02-28T12:00:11.047ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
9942025-02-28T12:00:11.047ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
9952025-02-28T12:00:11.047ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 285.229496ms iteration = 4 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
9962025-02-28T12:00:11.047ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
9972025-02-28T12:00:11.047ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9982025-02-28T12:00:11.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9992025-02-28T12:00:11.048ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 845, tv_nsec: 680801047 } })
10002025-02-28T12:00:11.048ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = service_zone_nat_tracker file = nexus/src/app/mod.rs:1155
10012025-02-28T12:00:11.057ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
10022025-02-28T12:00:11.057ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "199556a8-91c0-4154-9bb4-02e200e4037f", "content-length": "2", "date": "Fri, 28 Feb 2025 12:00:10 GMT"} })
10032025-02-28T12:00:11.057ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
10042025-02-28T12:00:11.058ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "962432e9-c696-46f1-9cf5-0ea689ee17ba", "content-length": "51", "date": "Fri, 28 Feb 2025 12:00:10 GMT"} })
10052025-02-28T12:00:11.058ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
10062025-02-28T12:00:11.060ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
10072025-02-28T12:00:11.090ZINFOnexus: assigned collector to new producer collector_id = a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79 file = nexus/src/app/oximeter.rs:76 producer_id = c1b34bc5-1c2f-4d3c-808c-b7a441aac295
10082025-02-28T12:00:11.090ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 55964 local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:59832 req_id = 1be80351-2ec9-4997-a08d-bc72b8c984b2 response_code = 201 uri = /metrics/producers
10092025-02-28T12:00:11.090ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "1be80351-2ec9-4997-a08d-bc72b8c984b2", "content-length": "41", "date": "Fri, 28 Feb 2025 12:00:11 GMT"} })
10102025-02-28T12:00:11.092ZDEBGnexus (producer-registration-task): registered with nexus successfully
10112025-02-28T12:00:11.093ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
10122025-02-28T12:00:11.109ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = service_zone_nat_tracker file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
10132025-02-28T12:00:11.109ZDEBGnexus (MgsClient): client request background_task = service_zone_nat_tracker body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
10142025-02-28T12:00:11.110ZDEBGnexus (MgsClient): client response background_task = service_zone_nat_tracker result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4e37dd2-a4a2-4cf1-952a-bff143425be5", "content-length": "26", "date": "Fri, 28 Feb 2025 12:00:10 GMT"} })
10152025-02-28T12:00:11.110ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = service_zone_nat_tracker file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
10162025-02-28T12:00:11.112ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = service_zone_nat_tracker file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
10172025-02-28T12:00:11.155ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
10182025-02-28T12:00:11.168ZDEBGnexus (DpdClient): client request background_task = service_zone_nat_tracker body = None method = POST uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/trigger
10192025-02-28T12:00:11.168ZDEBGnexus (DpdClient): client response background_task = service_zone_nat_tracker result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/trigger", status: 200, headers: {"content-type": "application/json", "x-request-id": "962b4663-7fe0-431d-978a-4edfcfa1b9ee", "content-length": "4", "date": "Fri, 28 Feb 2025 12:00:10 GMT"} })
10202025-02-28T12:00:11.169ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 706.911475ms iteration = 3 status = 4
10212025-02-28T12:00:11.198ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "fdf96c03-2198-4323-9d30-2016feeac152", "date": "Fri, 28 Feb 2025 12:00:10 GMT"} })
10222025-02-28T12:00:11.199ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
10232025-02-28T12:00:11.199ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
10242025-02-28T12:00:11.199ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e0a7acd-3d43-44c3-a2ec-fd183803ead1", "content-length": "691", "date": "Fri, 28 Feb 2025 12:00:11 GMT"} })
10252025-02-28T12:00:11.199ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "83b6c1e5-0219-4f0f-8ac7-afc896e6b66f", "content-length": "539", "date": "Fri, 28 Feb 2025 12:00:11 GMT"} })
10262025-02-28T12:00:11.199ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
10272025-02-28T12:00:11.199ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
10282025-02-28T12:00:11.199ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
10292025-02-28T12:00:11.201ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 738.437936ms iteration = 3 status = {}
10302025-02-28T12:00:11.201ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10312025-02-28T12:00:11.202ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10322025-02-28T12:00:11.211ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10332025-02-28T12:00:11.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10342025-02-28T12:00:11.216ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10352025-02-28T12:00:11.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10362025-02-28T12:00:11.218ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10372025-02-28T12:00:11.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10382025-02-28T12:00:11.225ZWARNnexus (BackgroundTasks): no bootstore config found in db background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1229 key = network_key rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
10392025-02-28T12:00:11.225ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10402025-02-28T12:00:11.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10412025-02-28T12:00:11.235ZINFOnexus (BackgroundTasks): updating bootstore config background_task = switch_port_config_manager config = EarlyNetworkConfig { body: EarlyNetworkConfigBody { ntp_servers: ["0.pool.ntp.org"], rack_network_config: Some(RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV2 { addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], autoneg: false, bgp_peers: [], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.2.100, rib_priority: None, vlan_id: None }], switch: Switch0, tx_eq: None, uplink_port_fec: Some(None), uplink_port_speed: Speed40G }], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 } }) }, generation: 2, schema_version: 2 } file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1286 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
10422025-02-28T12:00:11.235ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = PUT rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
10432025-02-28T12:00:11.238ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 204, headers: {"x-request-id": "de8c83b8-905b-4bd2-ad08-9b30eeb8aded", "date": "Fri, 28 Feb 2025 12:00:11 GMT"} })
10442025-02-28T12:00:11.238ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10452025-02-28T12:00:11.239ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10462025-02-28T12:00:11.246ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 783.835032ms iteration = 3 status = {}
10472025-02-28T12:00:20.460ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 8 reason = Timeout
10482025-02-28T12:00:20.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10492025-02-28T12:00:20.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10502025-02-28T12:00:20.462ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 8 reason = Timeout
10512025-02-28T12:00:20.462ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10522025-02-28T12:00:20.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10532025-02-28T12:00:20.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10542025-02-28T12:00:20.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10552025-02-28T12:00:20.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10562025-02-28T12:00:20.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10572025-02-28T12:00:20.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10582025-02-28T12:00:20.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10592025-02-28T12:00:20.470ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10602025-02-28T12:00:20.472ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
10612025-02-28T12:00:20.472ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10622025-02-28T12:00:20.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10632025-02-28T12:00:20.485ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
10642025-02-28T12:00:20.485ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.916212ms iteration = 8 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
10652025-02-28T12:00:20.500ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10662025-02-28T12:00:20.501ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10672025-02-28T12:00:20.503ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 40.116988ms iteration = 8 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10682025-02-28T12:00:24.257ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37015 req_id = a8bc5708-1ae8-432d-ac82-ca25058f513f roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
10692025-02-28T12:00:24.258ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37015 req_id = a8bc5708-1ae8-432d-ac82-ca25058f513f resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
10702025-02-28T12:00:24.260ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3290 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37015 req_id = a8bc5708-1ae8-432d-ac82-ca25058f513f response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
10712025-02-28T12:00:24.261ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37015 req_id = 5448f223-4ca0-4a76-a99e-a4625ab69e8e roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
10722025-02-28T12:00:24.261ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37015 req_id = 5448f223-4ca0-4a76-a99e-a4625ab69e8e resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
10732025-02-28T12:00:24.263ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2500 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37015 req_id = 5448f223-4ca0-4a76-a99e-a4625ab69e8e response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
10742025-02-28T12:00:30.460ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 9 reason = Timeout
10752025-02-28T12:00:30.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10762025-02-28T12:00:30.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10772025-02-28T12:00:30.462ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 9 reason = Timeout
10782025-02-28T12:00:30.462ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10792025-02-28T12:00:30.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10802025-02-28T12:00:30.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10812025-02-28T12:00:30.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10822025-02-28T12:00:30.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10832025-02-28T12:00:30.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10842025-02-28T12:00:30.468ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10852025-02-28T12:00:30.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10862025-02-28T12:00:30.470ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10872025-02-28T12:00:30.472ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
10882025-02-28T12:00:30.472ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10892025-02-28T12:00:30.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10902025-02-28T12:00:30.488ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
10912025-02-28T12:00:30.488ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 27.698948ms iteration = 9 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
10922025-02-28T12:00:30.493ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10932025-02-28T12:00:30.493ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10942025-02-28T12:00:30.498ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 36.056561ms iteration = 9 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10952025-02-28T12:00:34.264ZINFOnexus (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:56450 remote_addr = [fd00:1122:3344:101::d]:49165
10962025-02-28T12:00:34.271ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 7393 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 461dca84-210f-408c-b3e6-97d260f868f8 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
10972025-02-28T12:00:40.457ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 4 reason = Timeout
10982025-02-28T12:00:40.457ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
10992025-02-28T12:00:40.457ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 4 reason = Timeout
11002025-02-28T12:00:40.457ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
11012025-02-28T12:00:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11022025-02-28T12:00:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
11032025-02-28T12:00:40.460ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 10 reason = Timeout
11042025-02-28T12:00:40.460ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 4 reason = Timeout
11052025-02-28T12:00:40.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11062025-02-28T12:00:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11072025-02-28T12:00:40.461ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 875, tv_nsec: 93875358 } })
11082025-02-28T12:00:40.461ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
11092025-02-28T12:00:40.461ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 875, tv_nsec: 94513754 } })
11102025-02-28T12:00:40.461ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 4 reason = Timeout
11112025-02-28T12:00:40.462ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
11122025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 10 reason = Timeout
11132025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 4 reason = Timeout
11142025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11152025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11162025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 4 reason = Timeout
11172025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11182025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11192025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 4 reason = Timeout
11202025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
11212025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
11222025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 4 reason = Timeout
11232025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11242025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 4 reason = Timeout
11252025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
11262025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 4 reason = Timeout
11272025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
11282025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 5 reason = Timeout
11292025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
11302025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 4 reason = Timeout
11312025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11322025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11332025-02-28T12:00:40.462ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
11342025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11352025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11362025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11372025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
11382025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11392025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
11402025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11412025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
11422025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
11432025-02-28T12:00:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11442025-02-28T12:00:40.463ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 4 reason = Timeout
11452025-02-28T12:00:40.463ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
11462025-02-28T12:00:40.463ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 4 reason = Timeout
11472025-02-28T12:00:40.463ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
11482025-02-28T12:00:40.463ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 4 reason = Timeout
11492025-02-28T12:00:40.463ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
11502025-02-28T12:00:40.463ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 4 reason = Timeout
11512025-02-28T12:00:40.463ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
11522025-02-28T12:00:40.463ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 4 reason = Timeout
11532025-02-28T12:00:40.463ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
11542025-02-28T12:00:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
11552025-02-28T12:00:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
11562025-02-28T12:00:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
11572025-02-28T12:00:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
11582025-02-28T12:00:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
11592025-02-28T12:00:40.464ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.988436ms iteration = 4 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
11602025-02-28T12:00:40.464ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11612025-02-28T12:00:40.464ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11622025-02-28T12:00:40.465ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11632025-02-28T12:00:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
11642025-02-28T12:00:40.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11652025-02-28T12:00:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11662025-02-28T12:00:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11672025-02-28T12:00:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
11682025-02-28T12:00:40.466ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11692025-02-28T12:00:40.466ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
11702025-02-28T12:00:40.466ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11712025-02-28T12:00:40.466ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
11722025-02-28T12:00:40.466ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
11732025-02-28T12:00:40.466ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
11742025-02-28T12:00:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11752025-02-28T12:00:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11762025-02-28T12:00:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
11772025-02-28T12:00:40.467ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11782025-02-28T12:00:40.468ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11792025-02-28T12:00:40.469ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11802025-02-28T12:00:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
11812025-02-28T12:00:40.469ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11822025-02-28T12:00:40.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11832025-02-28T12:00:40.470ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
11842025-02-28T12:00:40.470ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
11852025-02-28T12:00:40.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
11862025-02-28T12:00:40.471ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11872025-02-28T12:00:40.471ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
11882025-02-28T12:00:40.471ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
11892025-02-28T12:00:40.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
11902025-02-28T12:00:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
11912025-02-28T12:00:40.473ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11922025-02-28T12:00:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11932025-02-28T12:00:40.476ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
11942025-02-28T12:00:40.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
11952025-02-28T12:00:40.478ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
11962025-02-28T12:00:40.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
11972025-02-28T12:00:40.478ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 15.167946ms iteration = 4 status = {"errors":[],"finish_invoked_ok":[]}
11982025-02-28T12:00:40.479ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11992025-02-28T12:00:40.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12002025-02-28T12:00:40.481ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 18.533228ms iteration = 4 status = {"errors":[],"requests_created_ok":[]}
12012025-02-28T12:00:40.483ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12022025-02-28T12:00:40.485ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12032025-02-28T12:00:40.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12042025-02-28T12:00:40.485ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12052025-02-28T12:00:40.486ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
12062025-02-28T12:00:40.486ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12072025-02-28T12:00:40.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12082025-02-28T12:00:40.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12092025-02-28T12:00:40.490ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 27.663007ms iteration = 4 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
12102025-02-28T12:00:40.492ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12112025-02-28T12:00:40.495ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
12122025-02-28T12:00:40.495ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12132025-02-28T12:00:40.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12142025-02-28T12:00:40.518ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
12152025-02-28T12:00:40.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
12162025-02-28T12:00:40.523ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12172025-02-28T12:00:40.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12182025-02-28T12:00:40.524ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.740985ms iteration = 10 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
12192025-02-28T12:00:40.524ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 61.610627ms iteration = 4 status = {"errors":[],"garbage_collect_requested":[]}
12202025-02-28T12:00:40.527ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12212025-02-28T12:00:40.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12222025-02-28T12:00:40.530ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12232025-02-28T12:00:40.531ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
12242025-02-28T12:00:40.531ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
12252025-02-28T12:00:40.531ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "20e5dc7c-c96d-463f-a642-cf5aeb87d645", "content-length": "26", "date": "Fri, 28 Feb 2025 12:00:40 GMT"} })
12262025-02-28T12:00:40.531ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
12272025-02-28T12:00:40.532ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
12282025-02-28T12:00:40.532ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
12292025-02-28T12:00:40.532ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
12302025-02-28T12:00:40.533ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
12312025-02-28T12:00:40.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
12322025-02-28T12:00:40.533ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
12332025-02-28T12:00:40.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
12342025-02-28T12:00:40.534ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
12352025-02-28T12:00:40.534ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12362025-02-28T12:00:40.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12372025-02-28T12:00:40.538ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 75.974374ms iteration = 4 status = 0
12382025-02-28T12:00:40.539ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
12392025-02-28T12:00:40.539ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 77.8709ms iteration = 4 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
12402025-02-28T12:00:40.541ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
12412025-02-28T12:00:40.541ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
12422025-02-28T12:00:40.541ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fab969c-dc63-41bf-ac5e-615784d16a67", "content-length": "26", "date": "Fri, 28 Feb 2025 12:00:40 GMT"} })
12432025-02-28T12:00:40.541ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
12442025-02-28T12:00:40.541ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12452025-02-28T12:00:40.542ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12462025-02-28T12:00:40.542ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
12472025-02-28T12:00:40.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12482025-02-28T12:00:40.544ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
12492025-02-28T12:00:40.544ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12502025-02-28T12:00:40.544ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12512025-02-28T12:00:40.544ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12522025-02-28T12:00:40.547ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
12532025-02-28T12:00:40.547ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12542025-02-28T12:00:40.547ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12552025-02-28T12:00:40.567ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 105.176199ms iteration = 4 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
12562025-02-28T12:00:40.577ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
12572025-02-28T12:00:40.577ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
12582025-02-28T12:00:40.577ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 114.982216ms iteration = 4 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
12592025-02-28T12:00:40.577ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 114.193737ms iteration = 4 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
12602025-02-28T12:00:40.578ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
12612025-02-28T12:00:40.578ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "af4ac2f5-585d-49db-ba15-4abd45c5646f", "content-length": "1", "date": "Fri, 28 Feb 2025 12:00:39 GMT"} })
12622025-02-28T12:00:40.580ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
12632025-02-28T12:00:40.580ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
12642025-02-28T12:00:40.583ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 125.828008ms iteration = 4 status = null
12652025-02-28T12:00:40.584ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
12662025-02-28T12:00:40.584ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 875, tv_nsec: 216827397 } })
12672025-02-28T12:00:40.584ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 122.068257ms iteration = 4 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
12682025-02-28T12:00:40.584ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
12692025-02-28T12:00:40.587ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 129.594859ms iteration = 4 status = {}
12702025-02-28T12:00:40.613ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12712025-02-28T12:00:40.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12722025-02-28T12:00:40.629ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 167.112164ms iteration = 5 status = {"physical_disks_added":0}
12732025-02-28T12:00:40.644ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
12742025-02-28T12:00:40.644ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
12752025-02-28T12:00:40.644ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b8cb6c6-9482-4c5d-847c-d3ec94516783", "content-length": "372", "date": "Fri, 28 Feb 2025 12:00:40 GMT"} })
12762025-02-28T12:00:40.644ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12772025-02-28T12:00:40.645ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12782025-02-28T12:00:40.653ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
12792025-02-28T12:00:40.653ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
12802025-02-28T12:00:40.653ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ede8180-fd29-43f9-a5c9-e4d27b50a378", "content-length": "26", "date": "Fri, 28 Feb 2025 12:00:40 GMT"} })
12812025-02-28T12:00:40.653ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
12822025-02-28T12:00:40.655ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
12832025-02-28T12:00:40.667ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
12842025-02-28T12:00:40.667ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 207.241102ms iteration = 10 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
12852025-02-28T12:00:40.761ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12862025-02-28T12:00:40.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12872025-02-28T12:00:40.763ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12882025-02-28T12:00:40.765ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
12892025-02-28T12:00:40.765ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12902025-02-28T12:00:40.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12912025-02-28T12:00:40.767ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12922025-02-28T12:00:40.770ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
12932025-02-28T12:00:40.770ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12942025-02-28T12:00:40.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12952025-02-28T12:00:40.792ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12962025-02-28T12:00:40.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12972025-02-28T12:00:40.795ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
12982025-02-28T12:00:40.795ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12992025-02-28T12:00:40.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13002025-02-28T12:00:40.868ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
13012025-02-28T12:00:40.869ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "52705e13-7956-4ea5-8116-4b3d0724727b", "content-length": "691", "date": "Fri, 28 Feb 2025 12:00:40 GMT"} })
13022025-02-28T12:00:40.869ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
13032025-02-28T12:00:40.869ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
13042025-02-28T12:00:40.869ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
13052025-02-28T12:00:40.870ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 408.332277ms iteration = 4 status = {}
13062025-02-28T12:00:40.894ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
13072025-02-28T12:00:40.894ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
13082025-02-28T12:00:40.894ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "8559fdae-0c94-41fc-96f4-e5fe10bae91e", "date": "Fri, 28 Feb 2025 12:00:40 GMT"} })
13092025-02-28T12:00:40.894ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13102025-02-28T12:00:40.895ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13112025-02-28T12:00:40.896ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13122025-02-28T12:00:40.897ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13132025-02-28T12:00:40.899ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13142025-02-28T12:00:40.900ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13152025-02-28T12:00:40.903ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
13162025-02-28T12:00:40.903ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
13172025-02-28T12:00:40.904ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "962775a0-8f8b-4d07-9b16-46a8cb6f501e", "date": "Fri, 28 Feb 2025 12:00:40 GMT"} })
13182025-02-28T12:00:40.905ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 443.28175ms iteration = 4 status = {}
13192025-02-28T12:00:40.913ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
13202025-02-28T12:00:40.913ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a5062f4-327a-4e12-9dbe-221e2d69d40e", "content-length": "134", "date": "Fri, 28 Feb 2025 12:00:40 GMT"} })
13212025-02-28T12:00:40.913ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
13222025-02-28T12:00:40.913ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
13232025-02-28T12:00:40.914ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
13242025-02-28T12:00:40.914ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
13252025-02-28T12:00:40.914ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13262025-02-28T12:00:40.914ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13272025-02-28T12:00:40.914ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13282025-02-28T12:00:40.914ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
13292025-02-28T12:00:40.914ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13302025-02-28T12:00:40.914ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
13312025-02-28T12:00:40.914ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
13322025-02-28T12:00:40.914ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13332025-02-28T12:00:40.914ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13342025-02-28T12:00:40.916ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
13352025-02-28T12:00:40.916ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "688b2549-535e-4ca0-bfcb-b1e99bea86e6", "content-length": "2", "date": "Fri, 28 Feb 2025 12:00:40 GMT"} })
13362025-02-28T12:00:40.916ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
13372025-02-28T12:00:40.916ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6f43340-5b7e-45ab-aa60-3e0815d9d86b", "content-length": "51", "date": "Fri, 28 Feb 2025 12:00:40 GMT"} })
13382025-02-28T12:00:40.916ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
13392025-02-28T12:00:40.916ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
13402025-02-28T12:00:41.025ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "8f751724-58e2-488d-8055-9c4d98836876", "date": "Fri, 28 Feb 2025 12:00:40 GMT"} })
13412025-02-28T12:00:41.025ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
13422025-02-28T12:00:41.025ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
13432025-02-28T12:00:41.026ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cab4e4f-0e32-4dd6-86b5-6b78c9b74eeb", "content-length": "682", "date": "Fri, 28 Feb 2025 12:00:41 GMT"} })
13442025-02-28T12:00:41.026ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13452025-02-28T12:00:41.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13462025-02-28T12:00:41.030ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13472025-02-28T12:00:41.030ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13482025-02-28T12:00:41.135ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13492025-02-28T12:00:41.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13502025-02-28T12:00:41.137ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13512025-02-28T12:00:41.137ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13522025-02-28T12:00:41.152ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 690.654103ms iteration = 4 status = {}
13532025-02-28T12:00:44.265ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 278 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = d65431d0-28b5-4779-8e97-0dae787d19ab response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
13542025-02-28T12:00:50.460ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 11 reason = Timeout
13552025-02-28T12:00:50.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13562025-02-28T12:00:50.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13572025-02-28T12:00:50.462ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 11 reason = Timeout
13582025-02-28T12:00:50.462ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13592025-02-28T12:00:50.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13602025-02-28T12:00:50.464ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13612025-02-28T12:00:50.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13622025-02-28T12:00:50.491ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13632025-02-28T12:00:50.494ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
13642025-02-28T12:00:50.494ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13652025-02-28T12:00:50.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13662025-02-28T12:00:50.497ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 35.836945ms iteration = 11 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
13672025-02-28T12:00:50.500ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13682025-02-28T12:00:50.500ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13692025-02-28T12:00:50.504ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13702025-02-28T12:00:50.507ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
13712025-02-28T12:00:50.507ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13722025-02-28T12:00:50.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13732025-02-28T12:00:50.528ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
13742025-02-28T12:00:50.528ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 68.087214ms iteration = 11 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
13752025-02-28T12:00:54.264ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 257 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 332bff5f-8291-4fe1-98d4-87a260c65e95 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
13762025-02-28T12:01:00.459ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 12 reason = Timeout
13772025-02-28T12:01:00.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13782025-02-28T12:01:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13792025-02-28T12:01:00.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 12 reason = Timeout
13802025-02-28T12:01:00.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13812025-02-28T12:01:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13822025-02-28T12:01:00.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13832025-02-28T12:01:00.461ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13842025-02-28T12:01:00.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13852025-02-28T12:01:00.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
13862025-02-28T12:01:00.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13872025-02-28T12:01:00.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13882025-02-28T12:01:00.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13892025-02-28T12:01:00.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13902025-02-28T12:01:00.468ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13912025-02-28T12:01:00.470ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
13922025-02-28T12:01:00.470ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13932025-02-28T12:01:00.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13942025-02-28T12:01:00.488ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
13952025-02-28T12:01:00.488ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 29.271717ms iteration = 12 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
13962025-02-28T12:01:00.501ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 41.732538ms iteration = 12 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
13972025-02-28T12:01:04.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 254 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 8159ca19-77e3-4916-91cd-2faa31d95727 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
13982025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 3 reason = Timeout
13992025-02-28T12:01:10.455ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-28 11:51:10.455510427 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
14002025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
14012025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 5 reason = Timeout
14022025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 3 reason = Timeout
14032025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 3 reason = Timeout
14042025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
14052025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
14062025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
14072025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
14082025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 5 reason = Timeout
14092025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
14102025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 5 reason = Timeout
14112025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 3 reason = Timeout
14122025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
14132025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 3 reason = Timeout
14142025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 3 reason = Timeout
14152025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
14162025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 5 reason = Timeout
14172025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
14182025-02-28T12:01:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
14192025-02-28T12:01:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
14202025-02-28T12:01:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
14212025-02-28T12:01:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
14222025-02-28T12:01:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
14232025-02-28T12:01:10.456ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5cefabf1-6831-46ff-98eb-1a3b5abad8b9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 905, tv_nsec: 90825825 } })
14242025-02-28T12:01:10.456ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5ebff2d9-48b6-4fe6-b852-684bb14b4955.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bddfb0f9-1d7b-45e4-b3a5-35aa6792956b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 905, tv_nsec: 91025730 } })
14252025-02-28T12:01:10.456ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
14262025-02-28T12:01:10.456ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.076156ms iteration = 3 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
14272025-02-28T12:01:10.456ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
14282025-02-28T12:01:10.456ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.26175ms iteration = 3 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
14292025-02-28T12:01:10.457ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
14302025-02-28T12:01:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
14312025-02-28T12:01:10.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 13 reason = Timeout
14322025-02-28T12:01:10.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 4 reason = Timeout
14332025-02-28T12:01:10.458ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
14342025-02-28T12:01:10.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14352025-02-28T12:01:10.458ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
14362025-02-28T12:01:10.458ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 8.741\u{b5}s iteration = 4 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
14372025-02-28T12:01:10.458ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 5 reason = Timeout
14382025-02-28T12:01:10.458ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-28 11:51:10.455510427 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
14392025-02-28T12:01:10.458ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.139736ms iteration = 3 status = {"expiration":"2025-02-28T11:51:10.455510427Z","pruned":[]}
14402025-02-28T12:01:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
14412025-02-28T12:01:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14422025-02-28T12:01:10.459ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 905, tv_nsec: 93594081 } })
14432025-02-28T12:01:10.459ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
14442025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 3 reason = Timeout
14452025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 5 reason = Timeout
14462025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 5 reason = Timeout
14472025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
14482025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
14492025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14502025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 5 reason = Timeout
14512025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
14522025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 5 reason = Timeout
14532025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 13 reason = Timeout
14542025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14552025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14562025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 5 reason = Timeout
14572025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 6 reason = Timeout
14582025-02-28T12:01:10.460ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
14592025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
14602025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 5 reason = Timeout
14612025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14622025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 3 reason = Timeout
14632025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
14642025-02-28T12:01:10.460ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
14652025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
14662025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 5 reason = Timeout
14672025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
14682025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
14692025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 5 reason = Timeout
14702025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14712025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14722025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14732025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
14742025-02-28T12:01:10.460ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 905, tv_nsec: 95572799 } })
14752025-02-28T12:01:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
14762025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
14772025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14782025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 5 reason = Timeout
14792025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
14802025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 5 reason = Timeout
14812025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
14822025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 5 reason = Timeout
14832025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
14842025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 5 reason = Timeout
14852025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
14862025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 3 reason = Timeout
14872025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
14882025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 3 reason = Timeout
14892025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
14902025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
14912025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 5 reason = Timeout
14922025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
14932025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
14942025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
14952025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14962025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14972025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14982025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
14992025-02-28T12:01:10.461ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
15002025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
15012025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
15022025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
15032025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15042025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
15052025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
15062025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
15072025-02-28T12:01:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
15082025-02-28T12:01:10.463ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
15092025-02-28T12:01:10.463ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
15102025-02-28T12:01:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
15112025-02-28T12:01:10.464ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.537863ms iteration = 5 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
15122025-02-28T12:01:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
15132025-02-28T12:01:10.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15142025-02-28T12:01:10.466ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
15152025-02-28T12:01:10.466ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
15162025-02-28T12:01:10.466ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
15172025-02-28T12:01:10.466ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
15182025-02-28T12:01:10.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
15192025-02-28T12:01:10.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15202025-02-28T12:01:10.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15212025-02-28T12:01:10.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15222025-02-28T12:01:10.469ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15232025-02-28T12:01:10.471ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15242025-02-28T12:01:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
15252025-02-28T12:01:10.471ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15262025-02-28T12:01:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15272025-02-28T12:01:10.472ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15282025-02-28T12:01:10.475ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
15292025-02-28T12:01:10.475ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15302025-02-28T12:01:10.475ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
15312025-02-28T12:01:10.475ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15322025-02-28T12:01:10.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15332025-02-28T12:01:10.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15342025-02-28T12:01:10.483ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 21.95555ms iteration = 5 status = {"errors":[],"garbage_collect_requested":[]}
15352025-02-28T12:01:10.486ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15362025-02-28T12:01:10.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15372025-02-28T12:01:10.487ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 25.937097ms iteration = 3 status = {"errors":[],"found_port_ok":[]}
15382025-02-28T12:01:10.487ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
15392025-02-28T12:01:10.487ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15402025-02-28T12:01:10.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
15412025-02-28T12:01:10.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15422025-02-28T12:01:10.493ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15432025-02-28T12:01:10.493ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15442025-02-28T12:01:10.498ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 905, tv_nsec: 132950372 } })
15452025-02-28T12:01:10.498ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
15462025-02-28T12:01:10.511ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15472025-02-28T12:01:10.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15482025-02-28T12:01:10.515ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15492025-02-28T12:01:10.516ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15502025-02-28T12:01:10.519ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
15512025-02-28T12:01:10.519ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
15522025-02-28T12:01:10.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
15532025-02-28T12:01:10.519ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15542025-02-28T12:01:10.521ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
15552025-02-28T12:01:10.521ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
15562025-02-28T12:01:10.522ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
15572025-02-28T12:01:10.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
15582025-02-28T12:01:10.523ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15592025-02-28T12:01:10.524ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15602025-02-28T12:01:10.524ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
15612025-02-28T12:01:10.524ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15622025-02-28T12:01:10.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
15632025-02-28T12:01:10.524ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15642025-02-28T12:01:10.525ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15652025-02-28T12:01:10.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15662025-02-28T12:01:10.526ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
15672025-02-28T12:01:10.526ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
15682025-02-28T12:01:10.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
15692025-02-28T12:01:10.527ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
15702025-02-28T12:01:10.527ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
15712025-02-28T12:01:10.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
15722025-02-28T12:01:10.527ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
15732025-02-28T12:01:10.527ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15742025-02-28T12:01:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15752025-02-28T12:01:10.528ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
15762025-02-28T12:01:10.528ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15772025-02-28T12:01:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15782025-02-28T12:01:10.529ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
15792025-02-28T12:01:10.529ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
15802025-02-28T12:01:10.529ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
15812025-02-28T12:01:10.529ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 69.18281ms iteration = 13 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
15822025-02-28T12:01:10.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
15832025-02-28T12:01:10.531ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 70.40812ms iteration = 5 status = {"errors":[],"finish_invoked_ok":[]}
15842025-02-28T12:01:10.531ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
15852025-02-28T12:01:10.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
15862025-02-28T12:01:10.534ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15872025-02-28T12:01:10.536ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15882025-02-28T12:01:10.536ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15892025-02-28T12:01:10.537ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
15902025-02-28T12:01:10.537ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
15912025-02-28T12:01:10.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
15922025-02-28T12:01:10.538ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
15932025-02-28T12:01:10.538ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
15942025-02-28T12:01:10.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
15952025-02-28T12:01:10.540ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
15962025-02-28T12:01:10.540ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
15972025-02-28T12:01:10.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
15982025-02-28T12:01:10.546ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
15992025-02-28T12:01:10.546ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
16002025-02-28T12:01:10.546ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 85.432582ms iteration = 3 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
16012025-02-28T12:01:10.549ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16022025-02-28T12:01:10.553ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
16032025-02-28T12:01:10.553ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16042025-02-28T12:01:10.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16052025-02-28T12:01:10.563ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
16062025-02-28T12:01:10.563ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 104.471202ms iteration = 13 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
16072025-02-28T12:01:10.568ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
16082025-02-28T12:01:10.568ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
16092025-02-28T12:01:10.575ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16102025-02-28T12:01:10.577ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
16112025-02-28T12:01:10.577ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16122025-02-28T12:01:10.578ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16132025-02-28T12:01:10.579ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
16142025-02-28T12:01:10.579ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16152025-02-28T12:01:10.579ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
16162025-02-28T12:01:10.579ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
16172025-02-28T12:01:10.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
16182025-02-28T12:01:10.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
16192025-02-28T12:01:10.583ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
16202025-02-28T12:01:10.583ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
16212025-02-28T12:01:10.585ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
16222025-02-28T12:01:10.585ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 125.236583ms iteration = 5 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
16232025-02-28T12:01:10.594ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
16242025-02-28T12:01:10.594ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
16252025-02-28T12:01:10.594ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 134.074326ms iteration = 3 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
16262025-02-28T12:01:10.629ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
16272025-02-28T12:01:10.629ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
16282025-02-28T12:01:10.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
16292025-02-28T12:01:10.630ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16302025-02-28T12:01:10.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16312025-02-28T12:01:10.631ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
16322025-02-28T12:01:10.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
16332025-02-28T12:01:10.643ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 182.358431ms iteration = 5 status = {"errors":[],"requests_created_ok":[]}
16342025-02-28T12:01:10.650ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
16352025-02-28T12:01:10.650ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16362025-02-28T12:01:10.650ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d322cd7-7047-4f48-a97d-abbb14e4e657", "content-length": "26", "date": "Fri, 28 Feb 2025 12:01:10 GMT"} })
16372025-02-28T12:01:10.650ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16382025-02-28T12:01:10.653ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
16392025-02-28T12:01:10.656ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16402025-02-28T12:01:10.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16412025-02-28T12:01:10.661ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
16422025-02-28T12:01:10.661ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
16432025-02-28T12:01:10.661ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
16442025-02-28T12:01:10.661ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
16452025-02-28T12:01:10.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
16462025-02-28T12:01:10.694ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
16472025-02-28T12:01:10.694ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 0 version = 2
16482025-02-28T12:01:10.694ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
16492025-02-28T12:01:10.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
16502025-02-28T12:01:10.747ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
16512025-02-28T12:01:10.747ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 286.781423ms iteration = 3 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
16522025-02-28T12:01:10.754ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
16532025-02-28T12:01:10.754ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16542025-02-28T12:01:10.754ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "45ed437d-4aa2-4b05-8bc6-3c95dcac5d2f", "content-length": "26", "date": "Fri, 28 Feb 2025 12:01:10 GMT"} })
16552025-02-28T12:01:10.754ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16562025-02-28T12:01:10.756ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
16572025-02-28T12:01:10.819ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
16582025-02-28T12:01:10.823ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
16592025-02-28T12:01:10.823ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
16602025-02-28T12:01:10.823ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "e23647a7-f54d-4f86-a214-79dcbf198de8", "content-length": "372", "date": "Fri, 28 Feb 2025 12:01:10 GMT"} })
16612025-02-28T12:01:10.823ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16622025-02-28T12:01:10.824ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16632025-02-28T12:01:10.828ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 367.119472ms iteration = 5 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
16642025-02-28T12:01:10.828ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
16652025-02-28T12:01:10.828ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16662025-02-28T12:01:10.829ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "07bfd2c9-27d4-43ae-988e-f8c905960d8d", "content-length": "26", "date": "Fri, 28 Feb 2025 12:01:10 GMT"} })
16672025-02-28T12:01:10.829ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16682025-02-28T12:01:10.831ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16692025-02-28T12:01:10.831ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16702025-02-28T12:01:10.832ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
16712025-02-28T12:01:10.832ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16722025-02-28T12:01:10.835ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
16732025-02-28T12:01:10.835ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
16742025-02-28T12:01:10.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
16752025-02-28T12:01:10.837ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
16762025-02-28T12:01:10.847ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 387.288199ms iteration = 6 status = {"physical_disks_added":0}
16772025-02-28T12:01:10.853ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 1 version = 2
16782025-02-28T12:01:10.853ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
16792025-02-28T12:01:10.853ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-02-28 11:59:52.542241 UTC dns_group = external generation = 2
16802025-02-28T12:01:10.853ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 398.0856ms iteration = 3 status = {"generation":2}
16812025-02-28T12:01:10.873ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 412.066158ms iteration = 5 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
16822025-02-28T12:01:10.879ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16832025-02-28T12:01:10.881ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
16842025-02-28T12:01:10.881ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
16852025-02-28T12:01:10.882ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
16862025-02-28T12:01:10.885ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 425.185164ms iteration = 5 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
16872025-02-28T12:01:10.886ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16882025-02-28T12:01:10.889ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
16892025-02-28T12:01:10.889ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16902025-02-28T12:01:10.889ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16912025-02-28T12:01:10.896ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16922025-02-28T12:01:10.896ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16932025-02-28T12:01:10.896ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
16942025-02-28T12:01:10.896ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 435.706188ms iteration = 5 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
16952025-02-28T12:01:10.899ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
16962025-02-28T12:01:10.899ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
16972025-02-28T12:01:10.899ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
16982025-02-28T12:01:10.906ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16992025-02-28T12:01:10.908ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
17002025-02-28T12:01:10.908ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17012025-02-28T12:01:10.908ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17022025-02-28T12:01:10.909ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
17032025-02-28T12:01:10.909ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
17042025-02-28T12:01:10.909ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "d4e743bb-dc50-44bf-ab84-a634a7539385", "date": "Fri, 28 Feb 2025 12:01:10 GMT"} })
17052025-02-28T12:01:10.909ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17062025-02-28T12:01:10.910ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17072025-02-28T12:01:10.911ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 450.67031ms iteration = 5 status = 0
17082025-02-28T12:01:10.911ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17092025-02-28T12:01:10.912ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17102025-02-28T12:01:10.922ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
17112025-02-28T12:01:10.922ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 461.754187ms iteration = 5 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
17122025-02-28T12:01:10.933ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-28T11:59:08.719399Z, time_modified: 2025-02-28T11:59:08.719399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }) }
17132025-02-28T12:01:10.933ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 478.03478ms iteration = 3 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]}},"default_endpoint":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
17142025-02-28T12:01:10.942ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
17152025-02-28T12:01:10.942ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
17162025-02-28T12:01:10.942ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17172025-02-28T12:01:10.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17182025-02-28T12:01:10.944ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17192025-02-28T12:01:10.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
17202025-02-28T12:01:10.947ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17212025-02-28T12:01:10.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17222025-02-28T12:01:10.950ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
17232025-02-28T12:01:10.950ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
17242025-02-28T12:01:10.950ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-02-28 11:59:51.915187 UTC dns_group = internal generation = 1
17252025-02-28T12:01:10.950ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 495.194464ms iteration = 3 status = {"generation":1}
17262025-02-28T12:01:10.974ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
17272025-02-28T12:01:10.974ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc54fee0-4dd4-48d7-a3a6-e045b6656616", "content-length": "1", "date": "Fri, 28 Feb 2025 12:01:10 GMT"} })
17282025-02-28T12:01:10.976ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
17292025-02-28T12:01:10.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
17302025-02-28T12:01:10.978ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 522.994396ms iteration = 5 status = null
17312025-02-28T12:01:10.991ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 535.63046ms iteration = 5 status = {}
17322025-02-28T12:01:10.994ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
17332025-02-28T12:01:10.995ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "70473767-56eb-40f0-82a7-ea91b61519f8", "date": "Fri, 28 Feb 2025 12:01:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
17342025-02-28T12:01:10.995ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
17352025-02-28T12:01:10.997ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "f3e1e4e0-ebe7-4d7a-bc24-6873200ca9dd", "date": "Fri, 28 Feb 2025 12:01:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
17362025-02-28T12:01:10.997ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
17372025-02-28T12:01:11.000ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "fbd7455c-6533-4189-b8e6-b04b7143b26b", "date": "Fri, 28 Feb 2025 12:01:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
17382025-02-28T12:01:11.000ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
17392025-02-28T12:01:11.002ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
17402025-02-28T12:01:11.002ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 546.662457ms iteration = 5 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
17412025-02-28T12:01:11.016ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17422025-02-28T12:01:11.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17432025-02-28T12:01:11.019ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
17442025-02-28T12:01:11.019ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
17452025-02-28T12:01:11.019ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "e7217934-e8c9-4b8c-bfd7-511014f7439a", "date": "Fri, 28 Feb 2025 12:01:10 GMT"} })
17462025-02-28T12:01:11.021ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 560.861ms iteration = 5 status = {}
17472025-02-28T12:01:11.033ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
17482025-02-28T12:01:11.034ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "0a6b6d66-e4f1-46be-8a7f-1f01a2e341e7", "date": "Fri, 28 Feb 2025 12:01:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
17492025-02-28T12:01:11.034ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
17502025-02-28T12:01:11.035ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "05030d0d-bc8c-4e7a-9b32-385643d9956d", "date": "Fri, 28 Feb 2025 12:01:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
17512025-02-28T12:01:11.035ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
17522025-02-28T12:01:11.037ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
17532025-02-28T12:01:11.037ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 582.152874ms iteration = 5 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
17542025-02-28T12:01:11.051ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
17552025-02-28T12:01:11.051ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "e02c1bc2-efc6-40e3-a086-4cf020b58299", "content-length": "691", "date": "Fri, 28 Feb 2025 12:01:10 GMT"} })
17562025-02-28T12:01:11.051ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
17572025-02-28T12:01:11.051ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
17582025-02-28T12:01:11.051ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
17592025-02-28T12:01:11.054ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 593.682572ms iteration = 5 status = {}
17602025-02-28T12:01:11.085ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17612025-02-28T12:01:11.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17622025-02-28T12:01:11.087ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
17632025-02-28T12:01:11.087ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17642025-02-28T12:01:11.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17652025-02-28T12:01:11.206ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
17662025-02-28T12:01:11.206ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "732cefea-2f9b-4f46-bd8c-dab842040d8b", "content-length": "134", "date": "Fri, 28 Feb 2025 12:01:10 GMT"} })
17672025-02-28T12:01:11.206ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
17682025-02-28T12:01:11.206ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
17692025-02-28T12:01:11.206ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
17702025-02-28T12:01:11.206ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
17712025-02-28T12:01:11.206ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17722025-02-28T12:01:11.206ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17732025-02-28T12:01:11.206ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17742025-02-28T12:01:11.206ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
17752025-02-28T12:01:11.207ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17762025-02-28T12:01:11.207ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
17772025-02-28T12:01:11.207ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
17782025-02-28T12:01:11.207ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17792025-02-28T12:01:11.207ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17802025-02-28T12:01:11.209ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
17812025-02-28T12:01:11.209ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd6b167e-6fc0-4f05-bfb8-a153f9766783", "content-length": "2", "date": "Fri, 28 Feb 2025 12:01:10 GMT"} })
17822025-02-28T12:01:11.209ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
17832025-02-28T12:01:11.209ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "68e88252-d14e-4f86-96e3-54252bafd416", "content-length": "51", "date": "Fri, 28 Feb 2025 12:01:10 GMT"} })
17842025-02-28T12:01:11.209ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
17852025-02-28T12:01:11.209ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
17862025-02-28T12:01:11.310ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "76ae2159-dfa6-4103-a840-47108d9a724e", "date": "Fri, 28 Feb 2025 12:01:10 GMT"} })
17872025-02-28T12:01:11.310ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
17882025-02-28T12:01:11.310ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
17892025-02-28T12:01:11.311ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c0cdbc8-9143-4876-be0b-9410c5591b9e", "content-length": "682", "date": "Fri, 28 Feb 2025 12:01:11 GMT"} })
17902025-02-28T12:01:11.311ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17912025-02-28T12:01:11.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17922025-02-28T12:01:11.315ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17932025-02-28T12:01:11.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17942025-02-28T12:01:11.430ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17952025-02-28T12:01:11.430ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17962025-02-28T12:01:11.432ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17972025-02-28T12:01:11.433ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17982025-02-28T12:01:11.441ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 980.429079ms iteration = 5 status = {}
17992025-02-28T12:01:14.263ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 325 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = c54d1c45-fe95-4bf2-81c1-20b6847188e2 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
18002025-02-28T12:01:20.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 14 reason = Timeout
18012025-02-28T12:01:20.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18022025-02-28T12:01:20.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18032025-02-28T12:01:20.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 14 reason = Timeout
18042025-02-28T12:01:20.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18052025-02-28T12:01:20.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18062025-02-28T12:01:20.498ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18072025-02-28T12:01:20.498ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18082025-02-28T12:01:20.498ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18092025-02-28T12:01:20.500ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 40.138369ms iteration = 14 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
18102025-02-28T12:01:20.500ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
18112025-02-28T12:01:20.500ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18122025-02-28T12:01:20.501ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18132025-02-28T12:01:20.504ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18142025-02-28T12:01:20.504ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18152025-02-28T12:01:20.533ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18162025-02-28T12:01:20.535ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
18172025-02-28T12:01:20.535ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18182025-02-28T12:01:20.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18192025-02-28T12:01:20.549ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
18202025-02-28T12:01:20.549ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 91.12233ms iteration = 14 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
18212025-02-28T12:01:24.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 275 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 3b1ce8bd-d493-4271-ba80-3b8efc8ea6fb response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
18222025-02-28T12:01:30.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 15 reason = Timeout
18232025-02-28T12:01:30.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18242025-02-28T12:01:30.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18252025-02-28T12:01:30.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 15 reason = Timeout
18262025-02-28T12:01:30.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18272025-02-28T12:01:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18282025-02-28T12:01:30.465ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18292025-02-28T12:01:30.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18302025-02-28T12:01:30.495ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18312025-02-28T12:01:30.497ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
18322025-02-28T12:01:30.497ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18332025-02-28T12:01:30.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18342025-02-28T12:01:30.500ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18352025-02-28T12:01:30.500ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18362025-02-28T12:01:30.502ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 42.143498ms iteration = 15 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
18372025-02-28T12:01:30.503ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18382025-02-28T12:01:30.506ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
18392025-02-28T12:01:30.506ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18402025-02-28T12:01:30.506ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18412025-02-28T12:01:30.521ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
18422025-02-28T12:01:30.521ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 63.442911ms iteration = 15 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
18432025-02-28T12:01:34.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 261 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 53d77623-e9d4-48be-abed-1007983e8c30 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
18442025-02-28T12:01:39.255ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::d]:43367
18452025-02-28T12:01:39.255ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43367 req_id = bcccfc03-bbfc-4593-9bfd-9375276e6fd8 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
18462025-02-28T12:01:39.256ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43367 req_id = bcccfc03-bbfc-4593-9bfd-9375276e6fd8 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
18472025-02-28T12:01:39.258ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3037 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43367 req_id = bcccfc03-bbfc-4593-9bfd-9375276e6fd8 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
18482025-02-28T12:01:39.259ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43367 req_id = 45d691a2-4f8f-4a88-bdf0-e0131fb7d1dc roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
18492025-02-28T12:01:39.259ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43367 req_id = 45d691a2-4f8f-4a88-bdf0-e0131fb7d1dc resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
18502025-02-28T12:01:39.363ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 104135 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43367 req_id = 45d691a2-4f8f-4a88-bdf0-e0131fb7d1dc response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
18512025-02-28T12:01:40.455ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 6 reason = Timeout
18522025-02-28T12:01:40.455ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 6 reason = Timeout
18532025-02-28T12:01:40.455ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
18542025-02-28T12:01:40.455ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
18552025-02-28T12:01:40.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
18562025-02-28T12:01:40.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
18572025-02-28T12:01:40.458ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 6 reason = Timeout
18582025-02-28T12:01:40.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 16 reason = Timeout
18592025-02-28T12:01:40.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18602025-02-28T12:01:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18612025-02-28T12:01:40.458ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 935, tv_nsec: 93631303 } })
18622025-02-28T12:01:40.459ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 935, tv_nsec: 93671973 } })
18632025-02-28T12:01:40.459ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
18642025-02-28T12:01:40.459ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
18652025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 16 reason = Timeout
18662025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 7 reason = Timeout
18672025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18682025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 6 reason = Timeout
18692025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18702025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18712025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 6 reason = Timeout
18722025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
18732025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 6 reason = Timeout
18742025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
18752025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 6 reason = Timeout
18762025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.313366ms iteration = 6 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
18772025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18782025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 6 reason = Timeout
18792025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
18802025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
18812025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 6 reason = Timeout
18822025-02-28T12:01:40.460ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
18832025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 6 reason = Timeout
18842025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18852025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 6 reason = Timeout
18862025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
18872025-02-28T12:01:40.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
18882025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18892025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18902025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18912025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
18922025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18932025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
18942025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
18952025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
18962025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
18972025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18982025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 6 reason = Timeout
18992025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 6 reason = Timeout
19002025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
19012025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
19022025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 6 reason = Timeout
19032025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
19042025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 6 reason = Timeout
19052025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
19062025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 6 reason = Timeout
19072025-02-28T12:01:40.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
19082025-02-28T12:01:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
19092025-02-28T12:01:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
19102025-02-28T12:01:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
19112025-02-28T12:01:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
19122025-02-28T12:01:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
19132025-02-28T12:01:40.463ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19142025-02-28T12:01:40.463ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19152025-02-28T12:01:40.463ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19162025-02-28T12:01:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
19172025-02-28T12:01:40.465ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
19182025-02-28T12:01:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
19192025-02-28T12:01:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
19202025-02-28T12:01:40.466ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19212025-02-28T12:01:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19222025-02-28T12:01:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
19232025-02-28T12:01:40.466ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19242025-02-28T12:01:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19252025-02-28T12:01:40.468ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19262025-02-28T12:01:40.470ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
19272025-02-28T12:01:40.470ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19282025-02-28T12:01:40.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19292025-02-28T12:01:40.474ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
19302025-02-28T12:01:40.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
19312025-02-28T12:01:40.475ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19322025-02-28T12:01:40.475ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19332025-02-28T12:01:40.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19342025-02-28T12:01:40.476ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 935, tv_nsec: 110710885 } })
19352025-02-28T12:01:40.476ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
19362025-02-28T12:01:40.477ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
19372025-02-28T12:01:40.477ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19382025-02-28T12:01:40.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19392025-02-28T12:01:40.510ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19402025-02-28T12:01:40.512ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
19412025-02-28T12:01:40.512ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19422025-02-28T12:01:40.512ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19432025-02-28T12:01:40.515ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19442025-02-28T12:01:40.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19452025-02-28T12:01:40.516ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19462025-02-28T12:01:40.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19472025-02-28T12:01:40.517ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 55.11373ms iteration = 6 status = {"errors":[],"garbage_collect_requested":[]}
19482025-02-28T12:01:40.518ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19492025-02-28T12:01:40.520ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19502025-02-28T12:01:40.521ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
19512025-02-28T12:01:40.521ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
19522025-02-28T12:01:40.521ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19532025-02-28T12:01:40.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
19542025-02-28T12:01:40.521ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 59.877435ms iteration = 6 status = {"errors":[],"finish_invoked_ok":[]}
19552025-02-28T12:01:40.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19562025-02-28T12:01:40.522ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
19572025-02-28T12:01:40.522ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
19582025-02-28T12:01:40.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
19592025-02-28T12:01:40.525ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
19602025-02-28T12:01:40.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
19612025-02-28T12:01:40.527ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
19622025-02-28T12:01:40.527ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
19632025-02-28T12:01:40.527ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
19642025-02-28T12:01:40.527ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
19652025-02-28T12:01:40.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
19662025-02-28T12:01:40.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
19672025-02-28T12:01:40.528ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19682025-02-28T12:01:40.529ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 67.95042ms iteration = 6 status = {"errors":[],"requests_created_ok":[]}
19692025-02-28T12:01:40.531ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
19702025-02-28T12:01:40.531ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
19712025-02-28T12:01:40.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
19722025-02-28T12:01:40.534ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19732025-02-28T12:01:40.535ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
19742025-02-28T12:01:40.535ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 76.564448ms iteration = 16 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
19752025-02-28T12:01:40.536ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
19762025-02-28T12:01:40.536ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
19772025-02-28T12:01:40.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
19782025-02-28T12:01:40.548ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
19792025-02-28T12:01:40.548ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19802025-02-28T12:01:40.548ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf979561-d9cf-4c26-9b48-74032fa7ca6d", "content-length": "26", "date": "Fri, 28 Feb 2025 12:01:40 GMT"} })
19812025-02-28T12:01:40.548ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19822025-02-28T12:01:40.550ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
19832025-02-28T12:01:40.554ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
19842025-02-28T12:01:40.554ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19852025-02-28T12:01:40.555ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7def995-ba41-4d1e-9372-869c37d82fff", "content-length": "26", "date": "Fri, 28 Feb 2025 12:01:40 GMT"} })
19862025-02-28T12:01:40.555ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19872025-02-28T12:01:40.556ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
19882025-02-28T12:01:40.559ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 98.971849ms iteration = 16 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
19892025-02-28T12:01:40.562ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 101.650414ms iteration = 6 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
19902025-02-28T12:01:40.568ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
19912025-02-28T12:01:40.568ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
19922025-02-28T12:01:40.570ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
19932025-02-28T12:01:40.570ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
19942025-02-28T12:01:40.570ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
19952025-02-28T12:01:40.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
19962025-02-28T12:01:40.573ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 111.782039ms iteration = 6 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
19972025-02-28T12:01:40.574ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
19982025-02-28T12:01:40.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
19992025-02-28T12:01:40.576ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20002025-02-28T12:01:40.578ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
20012025-02-28T12:01:40.578ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
20022025-02-28T12:01:40.578ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
20032025-02-28T12:01:40.579ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
20042025-02-28T12:01:40.579ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
20052025-02-28T12:01:40.579ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
20062025-02-28T12:01:40.580ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 119.139506ms iteration = 6 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
20072025-02-28T12:01:40.580ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8eb77388-ee5a-4300-8ce2-a2b50643c451", "content-length": "26", "date": "Fri, 28 Feb 2025 12:01:40 GMT"} })
20082025-02-28T12:01:40.580ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
20092025-02-28T12:01:40.581ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
20102025-02-28T12:01:40.581ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 120.616602ms iteration = 6 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
20112025-02-28T12:01:40.581ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
20122025-02-28T12:01:40.608ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 146.530017ms iteration = 6 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
20132025-02-28T12:01:40.623ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
20142025-02-28T12:01:40.624ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
20152025-02-28T12:01:40.624ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 163.336453ms iteration = 6 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
20162025-02-28T12:01:40.626ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20172025-02-28T12:01:40.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20182025-02-28T12:01:40.655ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 194.407353ms iteration = 7 status = {"physical_disks_added":0}
20192025-02-28T12:01:40.738ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20202025-02-28T12:01:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20212025-02-28T12:01:40.769ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20222025-02-28T12:01:40.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20232025-02-28T12:01:40.826ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
20242025-02-28T12:01:40.826ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "43227d69-2774-4018-9bb2-cc61e25a4cbc", "content-length": "1", "date": "Fri, 28 Feb 2025 12:01:40 GMT"} })
20252025-02-28T12:01:40.828ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
20262025-02-28T12:01:40.828ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
20272025-02-28T12:01:40.830ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 374.556542ms iteration = 6 status = {}
20282025-02-28T12:01:40.864ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
20292025-02-28T12:01:40.864ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
20302025-02-28T12:01:40.865ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "233f9446-bddf-432b-be91-08602519e473", "content-length": "372", "date": "Fri, 28 Feb 2025 12:01:40 GMT"} })
20312025-02-28T12:01:40.865ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20322025-02-28T12:01:40.865ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20332025-02-28T12:01:40.867ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20342025-02-28T12:01:40.867ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20352025-02-28T12:01:40.867ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20362025-02-28T12:01:40.867ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 412.16057ms iteration = 6 status = null
20372025-02-28T12:01:40.868ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
20382025-02-28T12:01:40.869ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20392025-02-28T12:01:40.869ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20402025-02-28T12:01:40.869ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20412025-02-28T12:01:40.870ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20422025-02-28T12:01:40.872ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
20432025-02-28T12:01:40.872ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20442025-02-28T12:01:40.872ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20452025-02-28T12:01:40.872ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
20462025-02-28T12:01:40.872ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20472025-02-28T12:01:40.872ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20482025-02-28T12:01:40.874ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 413.571314ms iteration = 6 status = 0
20492025-02-28T12:01:40.883ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20502025-02-28T12:01:40.883ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20512025-02-28T12:01:40.884ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
20522025-02-28T12:01:40.884ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
20532025-02-28T12:01:40.884ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "d830b32c-6611-4969-8205-9739041f583b", "date": "Fri, 28 Feb 2025 12:01:40 GMT"} })
20542025-02-28T12:01:40.884ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20552025-02-28T12:01:40.884ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20562025-02-28T12:01:40.886ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
20572025-02-28T12:01:40.886ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20582025-02-28T12:01:40.886ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20592025-02-28T12:01:40.887ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20602025-02-28T12:01:40.887ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20612025-02-28T12:01:40.890ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20622025-02-28T12:01:40.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20632025-02-28T12:01:40.896ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
20642025-02-28T12:01:40.896ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "2de6e4c9-656a-4015-bb7a-86bbef974e0a", "content-length": "134", "date": "Fri, 28 Feb 2025 12:01:40 GMT"} })
20652025-02-28T12:01:40.896ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
20662025-02-28T12:01:40.896ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
20672025-02-28T12:01:40.897ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
20682025-02-28T12:01:40.897ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
20692025-02-28T12:01:40.897ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
20702025-02-28T12:01:40.897ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
20712025-02-28T12:01:40.897ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
20722025-02-28T12:01:40.897ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
20732025-02-28T12:01:40.897ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
20742025-02-28T12:01:40.897ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
20752025-02-28T12:01:40.897ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
20762025-02-28T12:01:40.897ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20772025-02-28T12:01:40.897ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20782025-02-28T12:01:40.908ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
20792025-02-28T12:01:40.909ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e5d9535-6ad8-4eba-89ed-8d68b50e91cd", "content-length": "691", "date": "Fri, 28 Feb 2025 12:01:40 GMT"} })
20802025-02-28T12:01:40.909ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
20812025-02-28T12:01:40.909ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
20822025-02-28T12:01:40.909ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
20832025-02-28T12:01:40.909ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
20842025-02-28T12:01:40.909ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae83c8de-da86-4a7f-b226-b903c28078e0", "content-length": "2", "date": "Fri, 28 Feb 2025 12:01:40 GMT"} })
20852025-02-28T12:01:40.909ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
20862025-02-28T12:01:40.909ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d71dd08-2d38-411f-ade6-813d68ad6869", "content-length": "51", "date": "Fri, 28 Feb 2025 12:01:40 GMT"} })
20872025-02-28T12:01:40.909ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
20882025-02-28T12:01:40.909ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
20892025-02-28T12:01:40.910ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 449.87385ms iteration = 6 status = {}
20902025-02-28T12:01:40.957ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
20912025-02-28T12:01:40.957ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
20922025-02-28T12:01:41.013ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "1e79ebee-02ea-4269-9a3e-d1b0823e8ea8", "date": "Fri, 28 Feb 2025 12:01:40 GMT"} })
20932025-02-28T12:01:41.013ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
20942025-02-28T12:01:41.013ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
20952025-02-28T12:01:41.014ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7215f10-83c1-4b8c-8a49-06121ab35721", "content-length": "682", "date": "Fri, 28 Feb 2025 12:01:41 GMT"} })
20962025-02-28T12:01:41.014ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20972025-02-28T12:01:41.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20982025-02-28T12:01:41.014ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "b04cc6e1-93b6-4a79-8699-0354a73d29f8", "date": "Fri, 28 Feb 2025 12:01:40 GMT"} })
20992025-02-28T12:01:41.016ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 555.713305ms iteration = 6 status = {}
21002025-02-28T12:01:41.017ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21012025-02-28T12:01:41.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21022025-02-28T12:01:41.020ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21032025-02-28T12:01:41.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21042025-02-28T12:01:41.022ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21052025-02-28T12:01:41.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21062025-02-28T12:01:41.029ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 568.267458ms iteration = 6 status = {}
21072025-02-28T12:01:44.263ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 466 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 319ed26e-dfaa-40b7-91a3-b9941dc8d533 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
21082025-02-28T12:01:50.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 17 reason = Timeout
21092025-02-28T12:01:50.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21102025-02-28T12:01:50.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21112025-02-28T12:01:50.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 17 reason = Timeout
21122025-02-28T12:01:50.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21132025-02-28T12:01:50.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21142025-02-28T12:01:50.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21152025-02-28T12:01:50.462ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21162025-02-28T12:01:50.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21172025-02-28T12:01:50.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
21182025-02-28T12:01:50.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21192025-02-28T12:01:50.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21202025-02-28T12:01:50.464ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.896464ms iteration = 17 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
21212025-02-28T12:01:50.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21222025-02-28T12:01:50.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21232025-02-28T12:01:50.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21242025-02-28T12:01:50.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
21252025-02-28T12:01:50.469ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21262025-02-28T12:01:50.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21272025-02-28T12:01:50.484ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
21282025-02-28T12:01:50.485ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.318525ms iteration = 17 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
21292025-02-28T12:01:54.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 313 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 8ccd142b-a0e2-46cb-bae8-9b429fe39325 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
21302025-02-28T12:02:00.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 18 reason = Timeout
21312025-02-28T12:02:00.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21322025-02-28T12:02:00.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21332025-02-28T12:02:00.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 18 reason = Timeout
21342025-02-28T12:02:00.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21352025-02-28T12:02:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21362025-02-28T12:02:00.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21372025-02-28T12:02:00.462ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21382025-02-28T12:02:00.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21392025-02-28T12:02:00.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
21402025-02-28T12:02:00.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21412025-02-28T12:02:00.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21422025-02-28T12:02:00.464ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.089169ms iteration = 18 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
21432025-02-28T12:02:00.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21442025-02-28T12:02:00.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21452025-02-28T12:02:00.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21462025-02-28T12:02:00.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
21472025-02-28T12:02:00.469ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21482025-02-28T12:02:00.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21492025-02-28T12:02:00.478ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
21502025-02-28T12:02:00.478ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.852389ms iteration = 18 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
21512025-02-28T12:02:04.263ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 294 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 59709256-dd7d-4138-9896-7ad9d5e260fa response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
21522025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 4 reason = Timeout
21532025-02-28T12:02:10.455ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-28 11:52:10.455655523 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
21542025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 4 reason = Timeout
21552025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
21562025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
21572025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 7 reason = Timeout
21582025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21592025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
21602025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 6 reason = Timeout
21612025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 7 reason = Timeout
21622025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
21632025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 4 reason = Timeout
21642025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
21652025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 4 reason = Timeout
21662025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21672025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 6 reason = Timeout
21682025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21692025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 4 reason = Timeout
21702025-02-28T12:02:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 4 reason = Timeout
21712025-02-28T12:02:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
21722025-02-28T12:02:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21732025-02-28T12:02:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
21742025-02-28T12:02:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21752025-02-28T12:02:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21762025-02-28T12:02:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
21772025-02-28T12:02:10.456ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5cefabf1-6831-46ff-98eb-1a3b5abad8b9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 965, tv_nsec: 91095941 } })
21782025-02-28T12:02:10.456ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5ebff2d9-48b6-4fe6-b852-684bb14b4955.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bddfb0f9-1d7b-45e4-b3a5-35aa6792956b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 965, tv_nsec: 91286766 } })
21792025-02-28T12:02:10.456ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
21802025-02-28T12:02:10.456ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.036714ms iteration = 4 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
21812025-02-28T12:02:10.457ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
21822025-02-28T12:02:10.457ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.26532ms iteration = 4 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
21832025-02-28T12:02:10.457ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21842025-02-28T12:02:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21852025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 5 reason = Timeout
21862025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 19 reason = Timeout
21872025-02-28T12:02:10.458ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
21882025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 3 reason = Timeout
21892025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.34\u{b5}s iteration = 5 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
21902025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 7 reason = Timeout
21912025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21922025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
21932025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21942025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
21952025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
21962025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
21972025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
21982025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21992025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
22002025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
22012025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
22022025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
22032025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
22042025-02-28T12:02:10.458ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-28 11:52:10.455655523 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
22052025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.166336ms iteration = 4 status = {"expiration":"2025-02-28T11:52:10.455655523Z","pruned":[]}
22062025-02-28T12:02:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
22072025-02-28T12:02:10.459ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 965, tv_nsec: 94233057 } })
22082025-02-28T12:02:10.459ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
22092025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 7 reason = Timeout
22102025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 4 reason = Timeout
22112025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 7 reason = Timeout
22122025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
22132025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
22142025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
22152025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 7 reason = Timeout
22162025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
22172025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 8 reason = Timeout
22182025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22192025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 7 reason = Timeout
22202025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 7 reason = Timeout
22212025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22222025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 19 reason = Timeout
22232025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22242025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22252025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 7 reason = Timeout
22262025-02-28T12:02:10.460ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
22272025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22282025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 4 reason = Timeout
22292025-02-28T12:02:10.460ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
22302025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
22312025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 7 reason = Timeout
22322025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
22332025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 7 reason = Timeout
22342025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
22352025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
22362025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22372025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
22382025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
22392025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
22402025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22412025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22422025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22432025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22442025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22452025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22462025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
22472025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
22482025-02-28T12:02:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
22492025-02-28T12:02:10.460ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 965, tv_nsec: 95642621 } })
22502025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 7 reason = Timeout
22512025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
22522025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 7 reason = Timeout
22532025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
22542025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 4 reason = Timeout
22552025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 7 reason = Timeout
22562025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
22572025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
22582025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 7 reason = Timeout
22592025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
22602025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 7 reason = Timeout
22612025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
22622025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 4 reason = Timeout
22632025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
22642025-02-28T12:02:10.461ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
22652025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
22662025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
22672025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
22682025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
22692025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
22702025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
22712025-02-28T12:02:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
22722025-02-28T12:02:10.462ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
22732025-02-28T12:02:10.462ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
22742025-02-28T12:02:10.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
22752025-02-28T12:02:10.464ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22762025-02-28T12:02:10.464ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 6.397384ms iteration = 7 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
22772025-02-28T12:02:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
22782025-02-28T12:02:10.465ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
22792025-02-28T12:02:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
22802025-02-28T12:02:10.465ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22812025-02-28T12:02:10.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
22822025-02-28T12:02:10.467ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
22832025-02-28T12:02:10.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
22842025-02-28T12:02:10.468ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22852025-02-28T12:02:10.470ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22862025-02-28T12:02:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
22872025-02-28T12:02:10.470ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
22882025-02-28T12:02:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
22892025-02-28T12:02:10.471ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22902025-02-28T12:02:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
22912025-02-28T12:02:10.471ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
22922025-02-28T12:02:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
22932025-02-28T12:02:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
22942025-02-28T12:02:10.473ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22952025-02-28T12:02:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22962025-02-28T12:02:10.475ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22972025-02-28T12:02:10.478ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
22982025-02-28T12:02:10.478ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22992025-02-28T12:02:10.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23002025-02-28T12:02:10.478ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23012025-02-28T12:02:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
23022025-02-28T12:02:10.481ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
23032025-02-28T12:02:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
23042025-02-28T12:02:10.499ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 38.801487ms iteration = 7 status = {"errors":[],"requests_created_ok":[]}
23052025-02-28T12:02:10.502ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 41.274277ms iteration = 7 status = {"errors":[],"finish_invoked_ok":[]}
23062025-02-28T12:02:10.505ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 44.32859ms iteration = 4 status = {"errors":[],"found_port_ok":[]}
23072025-02-28T12:02:10.509ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
23082025-02-28T12:02:10.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
23092025-02-28T12:02:10.512ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23102025-02-28T12:02:10.514ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
23112025-02-28T12:02:10.514ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23122025-02-28T12:02:10.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23132025-02-28T12:02:10.518ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23142025-02-28T12:02:10.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23152025-02-28T12:02:10.518ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23162025-02-28T12:02:10.518ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23172025-02-28T12:02:10.520ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23182025-02-28T12:02:10.521ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23192025-02-28T12:02:10.522ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23202025-02-28T12:02:10.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23212025-02-28T12:02:10.523ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
23222025-02-28T12:02:10.523ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
23232025-02-28T12:02:10.523ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
23242025-02-28T12:02:10.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
23252025-02-28T12:02:10.523ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23262025-02-28T12:02:10.523ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23272025-02-28T12:02:10.524ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
23282025-02-28T12:02:10.524ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
23292025-02-28T12:02:10.524ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
23302025-02-28T12:02:10.524ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
23312025-02-28T12:02:10.525ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
23322025-02-28T12:02:10.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
23332025-02-28T12:02:10.525ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
23342025-02-28T12:02:10.525ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
23352025-02-28T12:02:10.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
23362025-02-28T12:02:10.525ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23372025-02-28T12:02:10.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23382025-02-28T12:02:10.526ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
23392025-02-28T12:02:10.526ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23402025-02-28T12:02:10.526ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
23412025-02-28T12:02:10.526ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
23422025-02-28T12:02:10.526ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
23432025-02-28T12:02:10.526ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23442025-02-28T12:02:10.526ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
23452025-02-28T12:02:10.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23462025-02-28T12:02:10.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
23472025-02-28T12:02:10.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23482025-02-28T12:02:10.527ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23492025-02-28T12:02:10.528ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
23502025-02-28T12:02:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
23512025-02-28T12:02:10.528ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23522025-02-28T12:02:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23532025-02-28T12:02:10.529ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23542025-02-28T12:02:10.530ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
23552025-02-28T12:02:10.530ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
23562025-02-28T12:02:10.530ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
23572025-02-28T12:02:10.531ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23582025-02-28T12:02:10.532ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
23592025-02-28T12:02:10.532ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
23602025-02-28T12:02:10.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
23612025-02-28T12:02:10.533ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
23622025-02-28T12:02:10.533ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23632025-02-28T12:02:10.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23642025-02-28T12:02:10.535ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
23652025-02-28T12:02:10.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
23662025-02-28T12:02:10.535ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 75.629886ms iteration = 19 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
23672025-02-28T12:02:10.536ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23682025-02-28T12:02:10.537ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
23692025-02-28T12:02:10.537ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
23702025-02-28T12:02:10.537ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
23712025-02-28T12:02:10.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
23722025-02-28T12:02:10.538ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
23732025-02-28T12:02:10.538ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23742025-02-28T12:02:10.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23752025-02-28T12:02:10.538ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 965, tv_nsec: 173647904 } })
23762025-02-28T12:02:10.539ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
23772025-02-28T12:02:10.540ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23782025-02-28T12:02:10.541ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 83.563847ms iteration = 3 status = {"nsuccess":5}
23792025-02-28T12:02:10.541ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
23802025-02-28T12:02:10.541ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
23812025-02-28T12:02:10.541ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 80.563594ms iteration = 4 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
23822025-02-28T12:02:10.543ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
23832025-02-28T12:02:10.543ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23842025-02-28T12:02:10.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23852025-02-28T12:02:10.550ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 89.125761ms iteration = 7 status = {"errors":[],"garbage_collect_requested":[]}
23862025-02-28T12:02:10.550ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 90.278349ms iteration = 8 status = {"physical_disks_added":0}
23872025-02-28T12:02:10.567ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23882025-02-28T12:02:10.567ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23892025-02-28T12:02:10.568ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23902025-02-28T12:02:10.568ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23912025-02-28T12:02:10.569ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23922025-02-28T12:02:10.570ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23932025-02-28T12:02:10.571ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23942025-02-28T12:02:10.571ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
23952025-02-28T12:02:10.571ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23962025-02-28T12:02:10.571ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23972025-02-28T12:02:10.576ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23982025-02-28T12:02:10.576ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
23992025-02-28T12:02:10.576ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
24002025-02-28T12:02:10.576ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
24012025-02-28T12:02:10.576ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
24022025-02-28T12:02:10.576ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
24032025-02-28T12:02:10.578ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
24042025-02-28T12:02:10.578ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24052025-02-28T12:02:10.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24062025-02-28T12:02:10.579ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24072025-02-28T12:02:10.579ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
24082025-02-28T12:02:10.579ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
24092025-02-28T12:02:10.579ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
24102025-02-28T12:02:10.579ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 119.525916ms iteration = 4 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
24112025-02-28T12:02:10.581ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 121.138674ms iteration = 7 status = 0
24122025-02-28T12:02:10.581ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
24132025-02-28T12:02:10.582ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
24142025-02-28T12:02:10.582ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
24152025-02-28T12:02:10.582ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
24162025-02-28T12:02:10.582ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
24172025-02-28T12:02:10.582ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
24182025-02-28T12:02:10.584ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24192025-02-28T12:02:10.585ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
24202025-02-28T12:02:10.585ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 0 version = 2
24212025-02-28T12:02:10.585ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
24222025-02-28T12:02:10.586ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
24232025-02-28T12:02:10.586ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
24242025-02-28T12:02:10.586ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
24252025-02-28T12:02:10.587ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
24262025-02-28T12:02:10.587ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
24272025-02-28T12:02:10.587ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 127.012556ms iteration = 7 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
24282025-02-28T12:02:10.587ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 127.144109ms iteration = 7 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
24292025-02-28T12:02:10.587ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24302025-02-28T12:02:10.589ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
24312025-02-28T12:02:10.590ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
24322025-02-28T12:02:10.590ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b62be2ae-6f34-463c-950c-92e2cd84530f", "content-length": "26", "date": "Fri, 28 Feb 2025 12:02:10 GMT"} })
24332025-02-28T12:02:10.590ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
24342025-02-28T12:02:10.591ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
24352025-02-28T12:02:10.591ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
24362025-02-28T12:02:10.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
24372025-02-28T12:02:10.592ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
24382025-02-28T12:02:10.594ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 1 version = 2
24392025-02-28T12:02:10.594ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
24402025-02-28T12:02:10.594ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-02-28 11:59:52.542241 UTC dns_group = external generation = 2
24412025-02-28T12:02:10.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 138.657537ms iteration = 4 status = {"generation":2}
24422025-02-28T12:02:10.597ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
24432025-02-28T12:02:10.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
24442025-02-28T12:02:10.598ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
24452025-02-28T12:02:10.598ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
24462025-02-28T12:02:10.598ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 137.788797ms iteration = 7 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
24472025-02-28T12:02:10.600ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
24482025-02-28T12:02:10.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
24492025-02-28T12:02:10.601ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24502025-02-28T12:02:10.603ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
24512025-02-28T12:02:10.603ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
24522025-02-28T12:02:10.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
24532025-02-28T12:02:10.609ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
24542025-02-28T12:02:10.609ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 149.669833ms iteration = 7 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
24552025-02-28T12:02:10.634ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
24562025-02-28T12:02:10.634ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
24572025-02-28T12:02:10.634ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
24582025-02-28T12:02:10.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
24592025-02-28T12:02:10.641ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24602025-02-28T12:02:10.643ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 182.217529ms iteration = 7 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
24612025-02-28T12:02:10.643ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24622025-02-28T12:02:10.644ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24632025-02-28T12:02:10.645ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
24642025-02-28T12:02:10.645ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
24652025-02-28T12:02:10.645ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
24662025-02-28T12:02:10.650ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
24672025-02-28T12:02:10.650ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
24682025-02-28T12:02:10.650ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-02-28 11:59:51.915187 UTC dns_group = internal generation = 1
24692025-02-28T12:02:10.650ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 194.483674ms iteration = 4 status = {"generation":1}
24702025-02-28T12:02:10.655ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24712025-02-28T12:02:10.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24722025-02-28T12:02:10.657ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24732025-02-28T12:02:10.661ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
24742025-02-28T12:02:10.661ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
24752025-02-28T12:02:10.661ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
24762025-02-28T12:02:10.661ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24772025-02-28T12:02:10.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24782025-02-28T12:02:10.661ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4dcc269-1945-4f6d-ae1d-7147ad81118d", "content-length": "26", "date": "Fri, 28 Feb 2025 12:02:10 GMT"} })
24792025-02-28T12:02:10.661ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
24802025-02-28T12:02:10.664ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
24812025-02-28T12:02:10.666ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
24822025-02-28T12:02:10.666ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
24832025-02-28T12:02:10.679ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
24842025-02-28T12:02:10.679ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 221.295292ms iteration = 19 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
24852025-02-28T12:02:10.743ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
24862025-02-28T12:02:10.743ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 283.545865ms iteration = 4 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
24872025-02-28T12:02:10.772ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
24882025-02-28T12:02:10.772ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
24892025-02-28T12:02:10.772ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "db9b294a-0d65-4d39-a1e5-ae89442dcc38", "content-length": "26", "date": "Fri, 28 Feb 2025 12:02:10 GMT"} })
24902025-02-28T12:02:10.772ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
24912025-02-28T12:02:10.774ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
24922025-02-28T12:02:10.778ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
24932025-02-28T12:02:10.779ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "5daec6eb-f513-477c-819f-f47130327295", "date": "Fri, 28 Feb 2025 12:02:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
24942025-02-28T12:02:10.779ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
24952025-02-28T12:02:10.781ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "fdeac50e-b08d-4d8e-bd09-3bf27093ebc6", "date": "Fri, 28 Feb 2025 12:02:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
24962025-02-28T12:02:10.781ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
24972025-02-28T12:02:10.782ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
24982025-02-28T12:02:10.782ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 327.037085ms iteration = 6 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
24992025-02-28T12:02:10.802ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-28T11:59:08.719399Z, time_modified: 2025-02-28T11:59:08.719399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }) }
25002025-02-28T12:02:10.802ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 346.815742ms iteration = 4 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]}},"default_endpoint":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
25012025-02-28T12:02:10.826ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
25022025-02-28T12:02:10.827ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "4470bd73-d5e2-46c6-98ab-bc76e63b0516", "content-length": "1", "date": "Fri, 28 Feb 2025 12:02:10 GMT"} })
25032025-02-28T12:02:10.828ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
25042025-02-28T12:02:10.829ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
25052025-02-28T12:02:10.907ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
25062025-02-28T12:02:11.016ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 560.660115ms iteration = 7 status = {}
25072025-02-28T12:02:11.030ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
25082025-02-28T12:02:11.030ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
25092025-02-28T12:02:11.031ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b12c761-3c37-453c-9d07-2b6b05e1f845", "content-length": "372", "date": "Fri, 28 Feb 2025 12:02:11 GMT"} })
25102025-02-28T12:02:11.031ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25112025-02-28T12:02:11.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25122025-02-28T12:02:11.032ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25132025-02-28T12:02:11.033ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 577.954962ms iteration = 7 status = null
25142025-02-28T12:02:11.034ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
25152025-02-28T12:02:11.034ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
25162025-02-28T12:02:11.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
25172025-02-28T12:02:11.063ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 602.635178ms iteration = 7 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
25182025-02-28T12:02:11.076ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
25192025-02-28T12:02:11.076ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
25202025-02-28T12:02:11.076ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "9f1ef4a7-c8c9-4b2e-9b01-038a34374734", "date": "Fri, 28 Feb 2025 12:02:11 GMT"} })
25212025-02-28T12:02:11.076ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25222025-02-28T12:02:11.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25232025-02-28T12:02:11.079ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25242025-02-28T12:02:11.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25252025-02-28T12:02:11.132ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
25262025-02-28T12:02:11.133ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "06071396-87ff-4d4b-ab3b-2a266a023221", "date": "Fri, 28 Feb 2025 12:02:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
25272025-02-28T12:02:11.133ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
25282025-02-28T12:02:11.134ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "fe8e4952-ae85-40ee-a801-952ec36ddfd9", "date": "Fri, 28 Feb 2025 12:02:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
25292025-02-28T12:02:11.134ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
25302025-02-28T12:02:11.137ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "0d12078b-1c77-4e8f-91d3-54f7890ea6fb", "date": "Fri, 28 Feb 2025 12:02:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
25312025-02-28T12:02:11.137ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
25322025-02-28T12:02:11.139ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
25332025-02-28T12:02:11.139ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 683.343557ms iteration = 6 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
25342025-02-28T12:02:11.153ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25352025-02-28T12:02:11.154ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25362025-02-28T12:02:11.155ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
25372025-02-28T12:02:11.155ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25382025-02-28T12:02:11.155ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25392025-02-28T12:02:11.161ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
25402025-02-28T12:02:11.168ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
25412025-02-28T12:02:11.168ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "6925de9e-0093-47d0-b2eb-51bb91aae37f", "content-length": "134", "date": "Fri, 28 Feb 2025 12:02:11 GMT"} })
25422025-02-28T12:02:11.168ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
25432025-02-28T12:02:11.168ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
25442025-02-28T12:02:11.169ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
25452025-02-28T12:02:11.169ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
25462025-02-28T12:02:11.169ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25472025-02-28T12:02:11.169ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25482025-02-28T12:02:11.169ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25492025-02-28T12:02:11.169ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
25502025-02-28T12:02:11.169ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25512025-02-28T12:02:11.169ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
25522025-02-28T12:02:11.169ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
25532025-02-28T12:02:11.169ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25542025-02-28T12:02:11.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25552025-02-28T12:02:11.172ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
25562025-02-28T12:02:11.172ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "5015e821-8fb3-4bb0-98b3-58636bd01e6c", "content-length": "2", "date": "Fri, 28 Feb 2025 12:02:11 GMT"} })
25572025-02-28T12:02:11.172ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
25582025-02-28T12:02:11.172ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "95903b7b-d3aa-4472-9363-8d95e81c87ee", "content-length": "51", "date": "Fri, 28 Feb 2025 12:02:11 GMT"} })
25592025-02-28T12:02:11.172ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
25602025-02-28T12:02:11.172ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
25612025-02-28T12:02:11.190ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25622025-02-28T12:02:11.190ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25632025-02-28T12:02:11.194ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
25642025-02-28T12:02:11.194ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
25652025-02-28T12:02:11.228ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "dc043baa-d2e3-4498-aea0-41f57737c3c7", "date": "Fri, 28 Feb 2025 12:02:10 GMT"} })
25662025-02-28T12:02:11.229ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 769.301721ms iteration = 7 status = {}
25672025-02-28T12:02:11.229ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "79eeb209-d0c7-469f-8113-eed38048bd6f", "content-length": "691", "date": "Fri, 28 Feb 2025 12:02:11 GMT"} })
25682025-02-28T12:02:11.229ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
25692025-02-28T12:02:11.229ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
25702025-02-28T12:02:11.229ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
25712025-02-28T12:02:11.231ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 770.914651ms iteration = 7 status = {}
25722025-02-28T12:02:11.322ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "3193c92f-c35c-45a3-ad0a-d2a32ca2f6a4", "date": "Fri, 28 Feb 2025 12:02:11 GMT"} })
25732025-02-28T12:02:11.322ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
25742025-02-28T12:02:11.322ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
25752025-02-28T12:02:11.322ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2a130a3-36be-4009-aa43-992a3d85fc7c", "content-length": "682", "date": "Fri, 28 Feb 2025 12:02:11 GMT"} })
25762025-02-28T12:02:11.322ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25772025-02-28T12:02:11.322ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25782025-02-28T12:02:11.326ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25792025-02-28T12:02:11.326ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25802025-02-28T12:02:11.329ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25812025-02-28T12:02:11.329ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25822025-02-28T12:02:11.331ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25832025-02-28T12:02:11.331ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25842025-02-28T12:02:11.339ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 878.854947ms iteration = 7 status = {}
25852025-02-28T12:02:14.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 294 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 8f830090-1650-41d9-98f9-181e5cb3180a response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
25862025-02-28T12:02:20.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 20 reason = Timeout
25872025-02-28T12:02:20.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25882025-02-28T12:02:20.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25892025-02-28T12:02:20.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 20 reason = Timeout
25902025-02-28T12:02:20.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25912025-02-28T12:02:20.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25922025-02-28T12:02:20.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25932025-02-28T12:02:20.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25942025-02-28T12:02:20.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25952025-02-28T12:02:20.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25962025-02-28T12:02:20.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25972025-02-28T12:02:20.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25982025-02-28T12:02:20.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25992025-02-28T12:02:20.470ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
26002025-02-28T12:02:20.470ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26012025-02-28T12:02:20.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26022025-02-28T12:02:20.491ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26032025-02-28T12:02:20.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26042025-02-28T12:02:20.492ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.812498ms iteration = 20 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
26052025-02-28T12:02:20.588ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
26062025-02-28T12:02:20.588ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 129.319642ms iteration = 20 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
26072025-02-28T12:02:24.256ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::d]:61935
26082025-02-28T12:02:24.256ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = 44aaddbb-2e2e-4bf3-abb7-9ea1685aa4e4 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
26092025-02-28T12:02:24.256ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = 44aaddbb-2e2e-4bf3-abb7-9ea1685aa4e4 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
26102025-02-28T12:02:24.259ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3157 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = 44aaddbb-2e2e-4bf3-abb7-9ea1685aa4e4 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
26112025-02-28T12:02:24.259ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = e77db17f-8057-4799-b28c-3ef0ad976f71 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
26122025-02-28T12:02:24.259ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = e77db17f-8057-4799-b28c-3ef0ad976f71 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
26132025-02-28T12:02:24.261ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2223 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = e77db17f-8057-4799-b28c-3ef0ad976f71 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
26142025-02-28T12:02:24.263ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 297 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 6cbdf7f8-703e-458a-9f64-6a242b621481 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
26152025-02-28T12:02:30.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 21 reason = Timeout
26162025-02-28T12:02:30.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26172025-02-28T12:02:30.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26182025-02-28T12:02:30.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 21 reason = Timeout
26192025-02-28T12:02:30.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26202025-02-28T12:02:30.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26212025-02-28T12:02:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26222025-02-28T12:02:30.462ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26232025-02-28T12:02:30.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26242025-02-28T12:02:30.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
26252025-02-28T12:02:30.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26262025-02-28T12:02:30.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26272025-02-28T12:02:30.464ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.833403ms iteration = 21 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
26282025-02-28T12:02:30.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26292025-02-28T12:02:30.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26302025-02-28T12:02:30.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26312025-02-28T12:02:30.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
26322025-02-28T12:02:30.469ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26332025-02-28T12:02:30.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26342025-02-28T12:02:30.477ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
26352025-02-28T12:02:30.478ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.697565ms iteration = 21 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
26362025-02-28T12:02:34.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 272 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 43b4178d-1698-4d27-9b16-80ae601b8eee response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
26372025-02-28T12:02:40.455ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 8 reason = Timeout
26382025-02-28T12:02:40.456ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
26392025-02-28T12:02:40.456ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 8 reason = Timeout
26402025-02-28T12:02:40.456ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
26412025-02-28T12:02:40.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
26422025-02-28T12:02:40.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
26432025-02-28T12:02:40.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 22 reason = Timeout
26442025-02-28T12:02:40.458ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 8 reason = Timeout
26452025-02-28T12:02:40.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26462025-02-28T12:02:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26472025-02-28T12:02:40.458ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 995, tv_nsec: 93402107 } })
26482025-02-28T12:02:40.458ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 995, tv_nsec: 93588752 } })
26492025-02-28T12:02:40.458ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
26502025-02-28T12:02:40.459ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
26512025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26522025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 22 reason = Timeout
26532025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 8 reason = Timeout
26542025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26552025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 8 reason = Timeout
26562025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26572025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
26582025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
26592025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 8 reason = Timeout
26602025-02-28T12:02:40.460ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
26612025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26622025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 9 reason = Timeout
26632025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
26642025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 8 reason = Timeout
26652025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 8 reason = Timeout
26662025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
26672025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
26682025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 8 reason = Timeout
26692025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
26702025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 8 reason = Timeout
26712025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26722025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 8 reason = Timeout
26732025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26742025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.207823ms iteration = 8 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
26752025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
26762025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26772025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26782025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
26792025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26802025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26812025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
26822025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
26832025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
26842025-02-28T12:02:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26852025-02-28T12:02:40.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 8 reason = Timeout
26862025-02-28T12:02:40.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 8 reason = Timeout
26872025-02-28T12:02:40.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
26882025-02-28T12:02:40.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
26892025-02-28T12:02:40.461ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 8 reason = Timeout
26902025-02-28T12:02:40.461ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
26912025-02-28T12:02:40.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 8 reason = Timeout
26922025-02-28T12:02:40.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
26932025-02-28T12:02:40.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 8 reason = Timeout
26942025-02-28T12:02:40.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
26952025-02-28T12:02:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
26962025-02-28T12:02:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
26972025-02-28T12:02:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
26982025-02-28T12:02:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
26992025-02-28T12:02:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
27002025-02-28T12:02:40.462ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27012025-02-28T12:02:40.462ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27022025-02-28T12:02:40.462ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27032025-02-28T12:02:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27042025-02-28T12:02:40.463ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
27052025-02-28T12:02:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
27062025-02-28T12:02:40.463ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
27072025-02-28T12:02:40.463ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
27082025-02-28T12:02:40.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27092025-02-28T12:02:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27102025-02-28T12:02:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
27112025-02-28T12:02:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
27122025-02-28T12:02:40.464ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
27132025-02-28T12:02:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
27142025-02-28T12:02:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
27152025-02-28T12:02:40.464ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27162025-02-28T12:02:40.464ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27172025-02-28T12:02:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27182025-02-28T12:02:40.465ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27192025-02-28T12:02:40.466ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27202025-02-28T12:02:40.466ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27212025-02-28T12:02:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
27222025-02-28T12:02:40.467ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27232025-02-28T12:02:40.467ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
27242025-02-28T12:02:40.467ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27252025-02-28T12:02:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27262025-02-28T12:02:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27272025-02-28T12:02:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
27282025-02-28T12:02:40.468ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27292025-02-28T12:02:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27302025-02-28T12:02:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
27312025-02-28T12:02:40.469ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27322025-02-28T12:02:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27332025-02-28T12:02:40.471ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 11.6157ms iteration = 22 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
27342025-02-28T12:02:40.473ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27352025-02-28T12:02:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27362025-02-28T12:02:40.474ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27372025-02-28T12:02:40.474ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27382025-02-28T12:02:40.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27392025-02-28T12:02:40.475ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 995, tv_nsec: 109742852 } })
27402025-02-28T12:02:40.475ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
27412025-02-28T12:02:40.476ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
27422025-02-28T12:02:40.476ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27432025-02-28T12:02:40.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27442025-02-28T12:02:40.476ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27452025-02-28T12:02:40.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27462025-02-28T12:02:40.483ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27472025-02-28T12:02:40.485ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
27482025-02-28T12:02:40.485ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27492025-02-28T12:02:40.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27502025-02-28T12:02:40.500ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 39.3508ms iteration = 8 status = {"errors":[],"finish_invoked_ok":[]}
27512025-02-28T12:02:40.513ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
27522025-02-28T12:02:40.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
27532025-02-28T12:02:40.515ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 54.177738ms iteration = 8 status = {"errors":[],"garbage_collect_requested":[]}
27542025-02-28T12:02:40.515ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
27552025-02-28T12:02:40.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
27562025-02-28T12:02:40.521ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
27572025-02-28T12:02:40.521ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
27582025-02-28T12:02:40.521ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
27592025-02-28T12:02:40.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
27602025-02-28T12:02:40.522ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 60.986623ms iteration = 8 status = {"errors":[],"requests_created_ok":[]}
27612025-02-28T12:02:40.522ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
27622025-02-28T12:02:40.522ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
27632025-02-28T12:02:40.523ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcf0cb37-ec0f-4805-960c-0774559f343c", "content-length": "26", "date": "Fri, 28 Feb 2025 12:02:40 GMT"} })
27642025-02-28T12:02:40.523ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
27652025-02-28T12:02:40.523ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
27662025-02-28T12:02:40.523ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
27672025-02-28T12:02:40.523ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 63.623476ms iteration = 8 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
27682025-02-28T12:02:40.524ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27692025-02-28T12:02:40.524ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
27702025-02-28T12:02:40.525ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
27712025-02-28T12:02:40.525ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
27722025-02-28T12:02:40.525ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0de86f44-1542-4814-8ca8-da276b6234c7", "content-length": "26", "date": "Fri, 28 Feb 2025 12:02:40 GMT"} })
27732025-02-28T12:02:40.525ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
27742025-02-28T12:02:40.526ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
27752025-02-28T12:02:40.526ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
27762025-02-28T12:02:40.526ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27772025-02-28T12:02:40.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27782025-02-28T12:02:40.535ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27792025-02-28T12:02:40.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27802025-02-28T12:02:40.540ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27812025-02-28T12:02:40.541ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
27822025-02-28T12:02:40.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
27832025-02-28T12:02:40.542ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
27842025-02-28T12:02:40.542ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27852025-02-28T12:02:40.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27862025-02-28T12:02:40.543ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
27872025-02-28T12:02:40.543ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 82.999784ms iteration = 8 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
27882025-02-28T12:02:40.546ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
27892025-02-28T12:02:40.546ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 87.803989ms iteration = 22 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
27902025-02-28T12:02:40.558ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 97.269308ms iteration = 8 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
27912025-02-28T12:02:40.560ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
27922025-02-28T12:02:40.560ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
27932025-02-28T12:02:40.560ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "83e60735-7e46-445e-afab-04aaf0616916", "content-length": "26", "date": "Fri, 28 Feb 2025 12:02:40 GMT"} })
27942025-02-28T12:02:40.560ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
27952025-02-28T12:02:40.562ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
27962025-02-28T12:02:40.574ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
27972025-02-28T12:02:40.574ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 114.006312ms iteration = 8 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
27982025-02-28T12:02:40.610ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 150.510263ms iteration = 9 status = {"physical_disks_added":0}
27992025-02-28T12:02:40.612ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 156.111929ms iteration = 8 status = {}
28002025-02-28T12:02:40.617ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
28012025-02-28T12:02:40.617ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4945260-5132-4494-9930-72c3d8e9b615", "content-length": "1", "date": "Fri, 28 Feb 2025 12:02:40 GMT"} })
28022025-02-28T12:02:40.619ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
28032025-02-28T12:02:40.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
28042025-02-28T12:02:40.652ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
28052025-02-28T12:02:40.652ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
28062025-02-28T12:02:40.653ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28072025-02-28T12:02:40.653ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c316d36-7b3b-49a3-afe8-a7a7e450c928", "content-length": "372", "date": "Fri, 28 Feb 2025 12:02:40 GMT"} })
28082025-02-28T12:02:40.653ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28092025-02-28T12:02:40.653ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28102025-02-28T12:02:40.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28112025-02-28T12:02:40.656ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28122025-02-28T12:02:40.656ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 201.019182ms iteration = 8 status = null
28132025-02-28T12:02:40.658ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
28142025-02-28T12:02:40.658ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28152025-02-28T12:02:40.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28162025-02-28T12:02:40.659ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28172025-02-28T12:02:40.660ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
28182025-02-28T12:02:40.660ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28192025-02-28T12:02:40.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28202025-02-28T12:02:40.714ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
28212025-02-28T12:02:40.714ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
28222025-02-28T12:02:40.714ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "2ba63f53-5455-413c-a019-a627dc1a9b95", "date": "Fri, 28 Feb 2025 12:02:40 GMT"} })
28232025-02-28T12:02:40.714ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28242025-02-28T12:02:40.714ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28252025-02-28T12:02:40.717ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28262025-02-28T12:02:40.717ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28272025-02-28T12:02:40.719ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28282025-02-28T12:02:40.720ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28292025-02-28T12:02:40.721ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
28302025-02-28T12:02:40.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
28312025-02-28T12:02:40.725ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
28322025-02-28T12:02:40.725ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
28332025-02-28T12:02:40.725ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "83458300-50f2-4da5-ab6b-e3dbe353492b", "date": "Fri, 28 Feb 2025 12:02:40 GMT"} })
28342025-02-28T12:02:40.727ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 267.642131ms iteration = 8 status = {}
28352025-02-28T12:02:40.730ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28362025-02-28T12:02:40.730ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28372025-02-28T12:02:40.732ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
28382025-02-28T12:02:40.732ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28392025-02-28T12:02:40.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28402025-02-28T12:02:40.736ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
28412025-02-28T12:02:40.736ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
28422025-02-28T12:02:40.773ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 311.725865ms iteration = 8 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
28432025-02-28T12:02:40.783ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
28442025-02-28T12:02:40.783ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 323.045668ms iteration = 8 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
28452025-02-28T12:02:40.783ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "df0e6491-0273-45d0-9c87-ce2ed38297d8", "content-length": "691", "date": "Fri, 28 Feb 2025 12:02:40 GMT"} })
28462025-02-28T12:02:40.783ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
28472025-02-28T12:02:40.783ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
28482025-02-28T12:02:40.783ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
28492025-02-28T12:02:40.785ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 324.724409ms iteration = 8 status = {}
28502025-02-28T12:02:40.818ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28512025-02-28T12:02:40.818ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28522025-02-28T12:02:40.820ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28532025-02-28T12:02:40.821ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28542025-02-28T12:02:40.822ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28552025-02-28T12:02:40.825ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
28562025-02-28T12:02:40.825ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28572025-02-28T12:02:40.825ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28582025-02-28T12:02:40.827ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 366.769854ms iteration = 8 status = 0
28592025-02-28T12:02:40.848ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
28602025-02-28T12:02:40.848ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a404e84-3f0c-4c2d-8234-e6625f05fb61", "content-length": "134", "date": "Fri, 28 Feb 2025 12:02:40 GMT"} })
28612025-02-28T12:02:40.848ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
28622025-02-28T12:02:40.848ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
28632025-02-28T12:02:40.848ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
28642025-02-28T12:02:40.848ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
28652025-02-28T12:02:40.848ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
28662025-02-28T12:02:40.848ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
28672025-02-28T12:02:40.848ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
28682025-02-28T12:02:40.848ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
28692025-02-28T12:02:40.849ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
28702025-02-28T12:02:40.849ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
28712025-02-28T12:02:40.849ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
28722025-02-28T12:02:40.849ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28732025-02-28T12:02:40.849ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28742025-02-28T12:02:40.877ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
28752025-02-28T12:02:40.877ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f8adc13-c6e2-48f4-b9d9-22a9d88ba0bb", "content-length": "2", "date": "Fri, 28 Feb 2025 12:02:40 GMT"} })
28762025-02-28T12:02:40.877ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
28772025-02-28T12:02:40.877ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb830e6e-abf5-42f5-b6a8-dc5a2858f965", "content-length": "51", "date": "Fri, 28 Feb 2025 12:02:40 GMT"} })
28782025-02-28T12:02:40.877ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
28792025-02-28T12:02:40.877ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
28802025-02-28T12:02:40.985ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "46d68c02-ff1d-4f5c-82d1-b5a4d8124e50", "date": "Fri, 28 Feb 2025 12:02:40 GMT"} })
28812025-02-28T12:02:40.985ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
28822025-02-28T12:02:40.985ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
28832025-02-28T12:02:40.986ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "b979b8ac-5998-47c5-8b19-0858e961b2cd", "content-length": "682", "date": "Fri, 28 Feb 2025 12:02:40 GMT"} })
28842025-02-28T12:02:40.986ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28852025-02-28T12:02:40.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28862025-02-28T12:02:40.990ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28872025-02-28T12:02:40.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28882025-02-28T12:02:40.992ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28892025-02-28T12:02:40.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28902025-02-28T12:02:40.994ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28912025-02-28T12:02:40.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28922025-02-28T12:02:41.002ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 542.578678ms iteration = 8 status = {}
28932025-02-28T12:02:41.091ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
28942025-02-28T12:02:41.091ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::a]:12221
28952025-02-28T12:02:41.122ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
28962025-02-28T12:02:41.122ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::a]:54249
28972025-02-28T12:02:41.122ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:54249 req_id = cf6d9ac1-d07c-41b5-826f-d8acba426eb7 roles = RoleSet { roles: {} } uri = /metrics/producers
28982025-02-28T12:02:41.122ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:54249 req_id = cf6d9ac1-d07c-41b5-826f-d8acba426eb7 resource = Database result = Ok(()) uri = /metrics/producers
28992025-02-28T12:02:41.165ZINFOnexus: assigned collector to new producer collector_id = a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79 file = nexus/src/app/oximeter.rs:76 producer_id = c1b34bc5-1c2f-4d3c-808c-b7a441aac295
29002025-02-28T12:02:41.165ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 43066 local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:54249 req_id = cf6d9ac1-d07c-41b5-826f-d8acba426eb7 response_code = 201 uri = /metrics/producers
29012025-02-28T12:02:41.165ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "cf6d9ac1-d07c-41b5-826f-d8acba426eb7", "content-length": "41", "date": "Fri, 28 Feb 2025 12:02:41 GMT"} })
29022025-02-28T12:02:41.167ZDEBGnexus (producer-registration-task): registered with nexus successfully
29032025-02-28T12:02:41.167ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
29042025-02-28T12:02:44.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 267 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = eaad5c11-a485-437d-8f16-2237cd1b5116 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
29052025-02-28T12:02:50.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 23 reason = Timeout
29062025-02-28T12:02:50.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29072025-02-28T12:02:50.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29082025-02-28T12:02:50.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 23 reason = Timeout
29092025-02-28T12:02:50.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29102025-02-28T12:02:50.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29112025-02-28T12:02:50.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29122025-02-28T12:02:50.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
29132025-02-28T12:02:50.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29142025-02-28T12:02:50.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29152025-02-28T12:02:50.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29162025-02-28T12:02:50.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29172025-02-28T12:02:50.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29182025-02-28T12:02:50.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
29192025-02-28T12:02:50.469ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29202025-02-28T12:02:50.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29212025-02-28T12:02:50.481ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
29222025-02-28T12:02:50.481ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.870163ms iteration = 23 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
29232025-02-28T12:02:50.490ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29242025-02-28T12:02:50.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29252025-02-28T12:02:50.491ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.981702ms iteration = 23 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
29262025-02-28T12:02:54.255ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = a24029c6-1912-4c32-82a2-528e2144632e roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
29272025-02-28T12:02:54.255ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = a24029c6-1912-4c32-82a2-528e2144632e resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
29282025-02-28T12:02:54.257ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2386 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = a24029c6-1912-4c32-82a2-528e2144632e response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
29292025-02-28T12:02:54.257ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = e669cbf2-7d29-4596-9ab7-82cf06757f88 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
29302025-02-28T12:02:54.257ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = e669cbf2-7d29-4596-9ab7-82cf06757f88 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
29312025-02-28T12:02:54.259ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2338 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = e669cbf2-7d29-4596-9ab7-82cf06757f88 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
29322025-02-28T12:02:54.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 248 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = daed44e6-2647-40fa-82f2-810dcffb2d16 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
29332025-02-28T12:03:00.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 24 reason = Timeout
29342025-02-28T12:03:00.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29352025-02-28T12:03:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29362025-02-28T12:03:00.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 24 reason = Timeout
29372025-02-28T12:03:00.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29382025-02-28T12:03:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29392025-02-28T12:03:00.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29402025-02-28T12:03:00.462ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29412025-02-28T12:03:00.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29422025-02-28T12:03:00.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
29432025-02-28T12:03:00.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29442025-02-28T12:03:00.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29452025-02-28T12:03:00.464ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.048747ms iteration = 24 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
29462025-02-28T12:03:00.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29472025-02-28T12:03:00.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29482025-02-28T12:03:00.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29492025-02-28T12:03:00.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
29502025-02-28T12:03:00.469ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29512025-02-28T12:03:00.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29522025-02-28T12:03:00.588ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
29532025-02-28T12:03:00.588ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 129.852924ms iteration = 24 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
29542025-02-28T12:03:04.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 441 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 722f3bf0-8776-4e58-a43b-8414f4c5b99e response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
29552025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 5 reason = Timeout
29562025-02-28T12:03:10.455ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-28 11:53:10.455144783 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
29572025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 5 reason = Timeout
29582025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
29592025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
29602025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
29612025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 7 reason = Timeout
29622025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 5 reason = Timeout
29632025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
29642025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
29652025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 9 reason = Timeout
29662025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
29672025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 5 reason = Timeout
29682025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 9 reason = Timeout
29692025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
29702025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 7 reason = Timeout
29712025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 5 reason = Timeout
29722025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
29732025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 5 reason = Timeout
29742025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
29752025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
29762025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
29772025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
29782025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
29792025-02-28T12:03:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
29802025-02-28T12:03:10.455ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5cefabf1-6831-46ff-98eb-1a3b5abad8b9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1025, tv_nsec: 90681891 } })
29812025-02-28T12:03:10.456ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5ebff2d9-48b6-4fe6-b852-684bb14b4955.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bddfb0f9-1d7b-45e4-b3a5-35aa6792956b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1025, tv_nsec: 90819445 } })
29822025-02-28T12:03:10.456ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
29832025-02-28T12:03:10.456ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
29842025-02-28T12:03:10.456ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.146458ms iteration = 5 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
29852025-02-28T12:03:10.456ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
29862025-02-28T12:03:10.456ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.193829ms iteration = 5 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
29872025-02-28T12:03:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
29882025-02-28T12:03:10.457ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
29892025-02-28T12:03:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
29902025-02-28T12:03:10.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 6 reason = Timeout
29912025-02-28T12:03:10.458ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
29922025-02-28T12:03:10.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 25 reason = Timeout
29932025-02-28T12:03:10.458ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 8.881\u{b5}s iteration = 6 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
29942025-02-28T12:03:10.458ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 9 reason = Timeout
29952025-02-28T12:03:10.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29962025-02-28T12:03:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29972025-02-28T12:03:10.458ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1025, tv_nsec: 93611242 } })
29982025-02-28T12:03:10.458ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29992025-02-28T12:03:10.458ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1025, tv_nsec: 93711784 } })
30002025-02-28T12:03:10.459ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
30012025-02-28T12:03:10.459ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
30022025-02-28T12:03:10.459ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30032025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
30042025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
30052025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30062025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 5 reason = Timeout
30072025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
30082025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 10 reason = Timeout
30092025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
30102025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 9 reason = Timeout
30112025-02-28T12:03:10.460ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
30122025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 9 reason = Timeout
30132025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30142025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
30152025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 9 reason = Timeout
30162025-02-28T12:03:10.460ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-28 11:53:10.455144783 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
30172025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
30182025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
30192025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.612296ms iteration = 5 status = {"expiration":"2025-02-28T11:53:10.455144783Z","pruned":[]}
30202025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 5 reason = Timeout
30212025-02-28T12:03:10.460ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
30222025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
30232025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 9 reason = Timeout
30242025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 9 reason = Timeout
30252025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
30262025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
30272025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
30282025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 9 reason = Timeout
30292025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
30302025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 9 reason = Timeout
30312025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 25 reason = Timeout
30322025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 9 reason = Timeout
30332025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30342025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
30352025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
30362025-02-28T12:03:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 9 reason = Timeout
30372025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
30382025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 9 reason = Timeout
30392025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
30402025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 9 reason = Timeout
30412025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
30422025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 9 reason = Timeout
30432025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
30442025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 9 reason = Timeout
30452025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
30462025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 5 reason = Timeout
30472025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
30482025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 5 reason = Timeout
30492025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
30502025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
30512025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
30522025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
30532025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30542025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
30552025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
30562025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30572025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
30582025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
30592025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
30602025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
30612025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
30622025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
30632025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
30642025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
30652025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
30662025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
30672025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
30682025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
30692025-02-28T12:03:10.461ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.576356ms iteration = 9 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
30702025-02-28T12:03:10.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30712025-02-28T12:03:10.463ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
30722025-02-28T12:03:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
30732025-02-28T12:03:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
30742025-02-28T12:03:10.464ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
30752025-02-28T12:03:10.464ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
30762025-02-28T12:03:10.464ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
30772025-02-28T12:03:10.464ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
30782025-02-28T12:03:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
30792025-02-28T12:03:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
30802025-02-28T12:03:10.465ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
30812025-02-28T12:03:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
30822025-02-28T12:03:10.465ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
30832025-02-28T12:03:10.465ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
30842025-02-28T12:03:10.465ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
30852025-02-28T12:03:10.465ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
30862025-02-28T12:03:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
30872025-02-28T12:03:10.465ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30882025-02-28T12:03:10.466ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
30892025-02-28T12:03:10.466ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
30902025-02-28T12:03:10.466ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
30912025-02-28T12:03:10.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
30922025-02-28T12:03:10.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30932025-02-28T12:03:10.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
30942025-02-28T12:03:10.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30952025-02-28T12:03:10.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
30962025-02-28T12:03:10.467ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
30972025-02-28T12:03:10.468ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30982025-02-28T12:03:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
30992025-02-28T12:03:10.468ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31002025-02-28T12:03:10.468ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31012025-02-28T12:03:10.469ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31022025-02-28T12:03:10.470ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
31032025-02-28T12:03:10.470ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31042025-02-28T12:03:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31052025-02-28T12:03:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
31062025-02-28T12:03:10.471ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
31072025-02-28T12:03:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
31082025-02-28T12:03:10.471ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31092025-02-28T12:03:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
31102025-02-28T12:03:10.471ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31112025-02-28T12:03:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
31122025-02-28T12:03:10.471ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31132025-02-28T12:03:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31142025-02-28T12:03:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31152025-02-28T12:03:10.472ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
31162025-02-28T12:03:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
31172025-02-28T12:03:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
31182025-02-28T12:03:10.473ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
31192025-02-28T12:03:10.473ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31202025-02-28T12:03:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
31212025-02-28T12:03:10.473ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31222025-02-28T12:03:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
31232025-02-28T12:03:10.473ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 13.101236ms iteration = 9 status = {"errors":[],"finish_invoked_ok":[]}
31242025-02-28T12:03:10.474ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
31252025-02-28T12:03:10.474ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
31262025-02-28T12:03:10.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
31272025-02-28T12:03:10.476ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 15.926884ms iteration = 9 status = {"errors":[],"garbage_collect_requested":[]}
31282025-02-28T12:03:10.477ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 16.468327ms iteration = 5 status = {"errors":[],"found_port_ok":[]}
31292025-02-28T12:03:10.478ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31302025-02-28T12:03:10.478ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31312025-02-28T12:03:10.479ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31322025-02-28T12:03:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
31332025-02-28T12:03:10.480ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
31342025-02-28T12:03:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
31352025-02-28T12:03:10.480ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
31362025-02-28T12:03:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
31372025-02-28T12:03:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
31382025-02-28T12:03:10.481ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
31392025-02-28T12:03:10.481ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
31402025-02-28T12:03:10.481ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
31412025-02-28T12:03:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
31422025-02-28T12:03:10.482ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
31432025-02-28T12:03:10.482ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
31442025-02-28T12:03:10.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
31452025-02-28T12:03:10.483ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31462025-02-28T12:03:10.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31472025-02-28T12:03:10.483ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31482025-02-28T12:03:10.483ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
31492025-02-28T12:03:10.484ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
31502025-02-28T12:03:10.485ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31512025-02-28T12:03:10.485ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
31522025-02-28T12:03:10.485ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31532025-02-28T12:03:10.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31542025-02-28T12:03:10.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31552025-02-28T12:03:10.486ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1025, tv_nsec: 121004663 } })
31562025-02-28T12:03:10.486ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
31572025-02-28T12:03:10.486ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31582025-02-28T12:03:10.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
31592025-02-28T12:03:10.488ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 27.728469ms iteration = 25 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
31602025-02-28T12:03:10.491ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31612025-02-28T12:03:10.491ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31622025-02-28T12:03:10.494ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
31632025-02-28T12:03:10.494ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
31642025-02-28T12:03:10.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
31652025-02-28T12:03:10.500ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
31662025-02-28T12:03:10.500ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31672025-02-28T12:03:10.500ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31682025-02-28T12:03:10.503ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
31692025-02-28T12:03:10.503ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 43.08827ms iteration = 9 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
31702025-02-28T12:03:10.522ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
31712025-02-28T12:03:10.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
31722025-02-28T12:03:10.522ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
31732025-02-28T12:03:10.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
31742025-02-28T12:03:10.524ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 63.617225ms iteration = 9 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
31752025-02-28T12:03:10.530ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
31762025-02-28T12:03:10.530ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
31772025-02-28T12:03:10.530ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 69.530428ms iteration = 5 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
31782025-02-28T12:03:10.531ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 70.265547ms iteration = 9 status = {"errors":[],"requests_created_ok":[]}
31792025-02-28T12:03:10.536ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 75.440311ms iteration = 9 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
31802025-02-28T12:03:10.536ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
31812025-02-28T12:03:10.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
31822025-02-28T12:03:10.538ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31832025-02-28T12:03:10.539ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
31842025-02-28T12:03:10.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
31852025-02-28T12:03:10.539ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
31862025-02-28T12:03:10.539ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
31872025-02-28T12:03:10.539ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 78.910105ms iteration = 5 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
31882025-02-28T12:03:10.541ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31892025-02-28T12:03:10.541ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
31902025-02-28T12:03:10.541ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
31912025-02-28T12:03:10.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
31922025-02-28T12:03:10.543ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
31932025-02-28T12:03:10.543ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
31942025-02-28T12:03:10.543ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
31952025-02-28T12:03:10.543ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 82.879341ms iteration = 9 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
31962025-02-28T12:03:10.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
31972025-02-28T12:03:10.547ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
31982025-02-28T12:03:10.547ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
31992025-02-28T12:03:10.547ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 86.59922ms iteration = 9 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
32002025-02-28T12:03:10.547ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
32012025-02-28T12:03:10.547ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
32022025-02-28T12:03:10.547ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
32032025-02-28T12:03:10.548ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
32042025-02-28T12:03:10.549ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32052025-02-28T12:03:10.553ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
32062025-02-28T12:03:10.553ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
32072025-02-28T12:03:10.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
32082025-02-28T12:03:10.553ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32092025-02-28T12:03:10.554ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32102025-02-28T12:03:10.554ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 93.675961ms iteration = 10 status = {"physical_disks_added":0}
32112025-02-28T12:03:10.555ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32122025-02-28T12:03:10.556ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
32132025-02-28T12:03:10.556ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
32142025-02-28T12:03:10.556ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-02-28 11:59:51.915187 UTC dns_group = internal generation = 1
32152025-02-28T12:03:10.556ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 101.204933ms iteration = 5 status = {"generation":1}
32162025-02-28T12:03:10.557ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
32172025-02-28T12:03:10.557ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 0 version = 2
32182025-02-28T12:03:10.557ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
32192025-02-28T12:03:10.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
32202025-02-28T12:03:10.558ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32212025-02-28T12:03:10.558ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
32222025-02-28T12:03:10.558ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32232025-02-28T12:03:10.559ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32242025-02-28T12:03:10.561ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
32252025-02-28T12:03:10.561ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
32262025-02-28T12:03:10.561ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
32272025-02-28T12:03:10.562ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32282025-02-28T12:03:10.563ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
32292025-02-28T12:03:10.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
32302025-02-28T12:03:10.563ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 1 version = 2
32312025-02-28T12:03:10.563ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
32322025-02-28T12:03:10.563ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-02-28 11:59:52.542241 UTC dns_group = external generation = 2
32332025-02-28T12:03:10.563ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 108.359076ms iteration = 5 status = {"generation":2}
32342025-02-28T12:03:10.564ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
32352025-02-28T12:03:10.564ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32362025-02-28T12:03:10.564ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32372025-02-28T12:03:10.565ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 104.920343ms iteration = 9 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
32382025-02-28T12:03:10.588ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-28T11:59:08.719399Z, time_modified: 2025-02-28T11:59:08.719399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }) }
32392025-02-28T12:03:10.588ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 133.604876ms iteration = 5 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]}},"default_endpoint":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
32402025-02-28T12:03:10.601ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
32412025-02-28T12:03:10.606ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
32422025-02-28T12:03:10.606ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 146.22529ms iteration = 5 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
32432025-02-28T12:03:10.611ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
32442025-02-28T12:03:10.616ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
32452025-02-28T12:03:10.616ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 158.837205ms iteration = 25 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
32462025-02-28T12:03:10.618ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
32472025-02-28T12:03:10.618ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
32482025-02-28T12:03:10.618ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b360f7c-9867-4bf1-9970-ef4ba3434b63", "content-length": "26", "date": "Fri, 28 Feb 2025 12:03:10 GMT"} })
32492025-02-28T12:03:10.618ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
32502025-02-28T12:03:10.620ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
32512025-02-28T12:03:10.747ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32522025-02-28T12:03:10.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32532025-02-28T12:03:10.754ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
32542025-02-28T12:03:10.754ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
32552025-02-28T12:03:10.755ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "751c56dc-6612-4848-aa83-cad82dcfa207", "content-length": "26", "date": "Fri, 28 Feb 2025 12:03:10 GMT"} })
32562025-02-28T12:03:10.755ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
32572025-02-28T12:03:10.755ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32582025-02-28T12:03:10.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32592025-02-28T12:03:10.757ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
32602025-02-28T12:03:10.763ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32612025-02-28T12:03:10.766ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
32622025-02-28T12:03:10.766ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32632025-02-28T12:03:10.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32642025-02-28T12:03:10.769ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
32652025-02-28T12:03:10.769ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
32662025-02-28T12:03:10.770ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a21c0dad-740a-4a2b-8354-8177e545fb0b", "content-length": "26", "date": "Fri, 28 Feb 2025 12:03:10 GMT"} })
32672025-02-28T12:03:10.770ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
32682025-02-28T12:03:10.772ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
32692025-02-28T12:03:10.838ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
32702025-02-28T12:03:10.839ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
32712025-02-28T12:03:10.839ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
32722025-02-28T12:03:10.840ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "72c318e4-4a87-4446-8033-1fa6ef94ccb6", "content-length": "372", "date": "Fri, 28 Feb 2025 12:03:10 GMT"} })
32732025-02-28T12:03:10.840ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32742025-02-28T12:03:10.840ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32752025-02-28T12:03:10.852ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
32762025-02-28T12:03:10.853ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "339851f5-a56b-4e88-b7cd-a706345528de", "date": "Fri, 28 Feb 2025 12:03:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
32772025-02-28T12:03:10.853ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
32782025-02-28T12:03:10.858ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "224fa60d-d198-4bf5-8019-daa560a75d43", "date": "Fri, 28 Feb 2025 12:03:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
32792025-02-28T12:03:10.858ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
32802025-02-28T12:03:10.860ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
32812025-02-28T12:03:10.860ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 405.509389ms iteration = 7 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
32822025-02-28T12:03:10.904ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
32832025-02-28T12:03:10.904ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "877f3096-b172-4789-b59e-2efbae3144ee", "content-length": "1", "date": "Fri, 28 Feb 2025 12:03:10 GMT"} })
32842025-02-28T12:03:10.915ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
32852025-02-28T12:03:10.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
32862025-02-28T12:03:10.941ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 486.647338ms iteration = 9 status = {}
32872025-02-28T12:03:10.944ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
32882025-02-28T12:03:10.944ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "44050f07-1bfb-422c-8c71-d20ad75c9906", "content-length": "691", "date": "Fri, 28 Feb 2025 12:03:10 GMT"} })
32892025-02-28T12:03:10.944ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
32902025-02-28T12:03:10.944ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
32912025-02-28T12:03:10.944ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
32922025-02-28T12:03:10.946ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 485.950171ms iteration = 9 status = 0
32932025-02-28T12:03:10.947ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 486.903424ms iteration = 9 status = {}
32942025-02-28T12:03:10.952ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 496.909375ms iteration = 9 status = null
32952025-02-28T12:03:10.963ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
32962025-02-28T12:03:10.963ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
32972025-02-28T12:03:10.963ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "a7dcc2cb-8fc2-4d97-8f05-c57c89953a5b", "date": "Fri, 28 Feb 2025 12:03:10 GMT"} })
32982025-02-28T12:03:10.963ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32992025-02-28T12:03:10.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33002025-02-28T12:03:10.966ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33012025-02-28T12:03:10.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33022025-02-28T12:03:10.968ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33032025-02-28T12:03:10.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33042025-02-28T12:03:10.973ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
33052025-02-28T12:03:10.973ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
33062025-02-28T12:03:10.973ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
33072025-02-28T12:03:10.973ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "1d43d7fa-aed6-4921-94bd-728eec85c3d2", "date": "Fri, 28 Feb 2025 12:03:10 GMT"} })
33082025-02-28T12:03:10.974ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "f5358f9f-6141-4ab4-af4a-5de60efd382c", "date": "Fri, 28 Feb 2025 12:03:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
33092025-02-28T12:03:10.974ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
33102025-02-28T12:03:10.975ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 515.166666ms iteration = 9 status = {}
33112025-02-28T12:03:10.976ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "5e475bb2-75eb-4462-ab8a-275d43b3f520", "date": "Fri, 28 Feb 2025 12:03:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
33122025-02-28T12:03:10.976ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
33132025-02-28T12:03:10.978ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "3a9ee00d-d32e-4755-bc25-6efb04a675f6", "date": "Fri, 28 Feb 2025 12:03:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
33142025-02-28T12:03:10.978ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
33152025-02-28T12:03:10.980ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
33162025-02-28T12:03:10.980ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 525.013584ms iteration = 7 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
33172025-02-28T12:03:11.019ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33182025-02-28T12:03:11.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33192025-02-28T12:03:11.022ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
33202025-02-28T12:03:11.022ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33212025-02-28T12:03:11.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33222025-02-28T12:03:11.030ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
33232025-02-28T12:03:11.030ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a8f7510-e13b-4310-bfd3-f47dddaf2bbd", "content-length": "134", "date": "Fri, 28 Feb 2025 12:03:11 GMT"} })
33242025-02-28T12:03:11.030ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
33252025-02-28T12:03:11.030ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
33262025-02-28T12:03:11.032ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
33272025-02-28T12:03:11.032ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
33282025-02-28T12:03:11.032ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33292025-02-28T12:03:11.032ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33302025-02-28T12:03:11.032ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33312025-02-28T12:03:11.032ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
33322025-02-28T12:03:11.032ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33332025-02-28T12:03:11.032ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
33342025-02-28T12:03:11.032ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
33352025-02-28T12:03:11.032ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33362025-02-28T12:03:11.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33372025-02-28T12:03:11.035ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
33382025-02-28T12:03:11.035ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "49a23af4-e758-4379-91f7-fa4c66c9931f", "content-length": "2", "date": "Fri, 28 Feb 2025 12:03:11 GMT"} })
33392025-02-28T12:03:11.035ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
33402025-02-28T12:03:11.035ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "a28ad3ab-ba8a-4a7d-a192-42e9c706cfa9", "content-length": "51", "date": "Fri, 28 Feb 2025 12:03:11 GMT"} })
33412025-02-28T12:03:11.035ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
33422025-02-28T12:03:11.035ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
33432025-02-28T12:03:11.130ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "abf1c27d-aa95-443c-aed5-f3d7e14824ba", "date": "Fri, 28 Feb 2025 12:03:10 GMT"} })
33442025-02-28T12:03:11.130ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
33452025-02-28T12:03:11.130ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
33462025-02-28T12:03:11.130ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "1eb0c043-5011-4d37-abd2-7047cad64111", "content-length": "682", "date": "Fri, 28 Feb 2025 12:03:11 GMT"} })
33472025-02-28T12:03:11.131ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33482025-02-28T12:03:11.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33492025-02-28T12:03:11.134ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33502025-02-28T12:03:11.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33512025-02-28T12:03:11.139ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33522025-02-28T12:03:11.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33532025-02-28T12:03:11.142ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33542025-02-28T12:03:11.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33552025-02-28T12:03:11.149ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 688.856ms iteration = 9 status = {}
33562025-02-28T12:03:14.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 291 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 4328d106-27bd-49af-8316-589de887ff2a response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
33572025-02-28T12:03:20.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 26 reason = Timeout
33582025-02-28T12:03:20.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33592025-02-28T12:03:20.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33602025-02-28T12:03:20.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 26 reason = Timeout
33612025-02-28T12:03:20.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33622025-02-28T12:03:20.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33632025-02-28T12:03:20.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33642025-02-28T12:03:20.461ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33652025-02-28T12:03:20.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33662025-02-28T12:03:20.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
33672025-02-28T12:03:20.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33682025-02-28T12:03:20.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33692025-02-28T12:03:20.463ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.023197ms iteration = 26 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
33702025-02-28T12:03:20.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33712025-02-28T12:03:20.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33722025-02-28T12:03:20.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33732025-02-28T12:03:20.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
33742025-02-28T12:03:20.469ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33752025-02-28T12:03:20.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33762025-02-28T12:03:20.478ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
33772025-02-28T12:03:20.478ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.927021ms iteration = 26 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
33782025-02-28T12:03:24.255ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = 1c15c3e4-44ec-431c-9b1c-b94291a2ad75 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
33792025-02-28T12:03:24.255ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = 1c15c3e4-44ec-431c-9b1c-b94291a2ad75 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
33802025-02-28T12:03:24.258ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2975 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = 1c15c3e4-44ec-431c-9b1c-b94291a2ad75 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
33812025-02-28T12:03:24.258ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = 342cd823-2bfe-4c4f-949d-23e74f104743 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
33822025-02-28T12:03:24.259ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = 342cd823-2bfe-4c4f-949d-23e74f104743 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
33832025-02-28T12:03:24.261ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2784 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = 342cd823-2bfe-4c4f-949d-23e74f104743 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
33842025-02-28T12:03:24.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 230 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 1ad9072d-ef66-4b3e-9c9a-edbf44437c8d response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
33852025-02-28T12:03:30.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 27 reason = Timeout
33862025-02-28T12:03:30.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33872025-02-28T12:03:30.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33882025-02-28T12:03:30.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 27 reason = Timeout
33892025-02-28T12:03:30.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33902025-02-28T12:03:30.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33912025-02-28T12:03:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33922025-02-28T12:03:30.462ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33932025-02-28T12:03:30.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33942025-02-28T12:03:30.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
33952025-02-28T12:03:30.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33962025-02-28T12:03:30.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33972025-02-28T12:03:30.464ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.906994ms iteration = 27 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
33982025-02-28T12:03:30.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33992025-02-28T12:03:30.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34002025-02-28T12:03:30.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34012025-02-28T12:03:30.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
34022025-02-28T12:03:30.469ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34032025-02-28T12:03:30.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34042025-02-28T12:03:30.479ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
34052025-02-28T12:03:30.479ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.436997ms iteration = 27 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
34062025-02-28T12:03:34.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 296 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 6b860743-2f22-4cb6-8a56-c3f757e5686a response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
34072025-02-28T12:03:40.454ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 10 reason = Timeout
34082025-02-28T12:03:40.454ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
34092025-02-28T12:03:40.454ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 10 reason = Timeout
34102025-02-28T12:03:40.454ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
34112025-02-28T12:03:40.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
34122025-02-28T12:03:40.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
34132025-02-28T12:03:40.457ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1055, tv_nsec: 92164997 } })
34142025-02-28T12:03:40.457ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
34152025-02-28T12:03:40.457ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1055, tv_nsec: 92648589 } })
34162025-02-28T12:03:40.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 28 reason = Timeout
34172025-02-28T12:03:40.457ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 10 reason = Timeout
34182025-02-28T12:03:40.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34192025-02-28T12:03:40.457ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
34202025-02-28T12:03:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34212025-02-28T12:03:40.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34222025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 28 reason = Timeout
34232025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34242025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 10 reason = Timeout
34252025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
34262025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 10 reason = Timeout
34272025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
34282025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
34292025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 10 reason = Timeout
34302025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
34312025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 10 reason = Timeout
34322025-02-28T12:03:40.460ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
34332025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 10 reason = Timeout
34342025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34352025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
34362025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 10 reason = Timeout
34372025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34382025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 11 reason = Timeout
34392025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 10 reason = Timeout
34402025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
34412025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 10 reason = Timeout
34422025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34432025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34442025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34452025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
34462025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
34472025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
34482025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
34492025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
34502025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34512025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34522025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34532025-02-28T12:03:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34542025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.412833ms iteration = 10 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
34552025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 10 reason = Timeout
34562025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 10 reason = Timeout
34572025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
34582025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
34592025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 10 reason = Timeout
34602025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
34612025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 10 reason = Timeout
34622025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
34632025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 10 reason = Timeout
34642025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
34652025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
34662025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
34672025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
34682025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
34692025-02-28T12:03:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
34702025-02-28T12:03:40.462ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34712025-02-28T12:03:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
34722025-02-28T12:03:40.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34732025-02-28T12:03:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34742025-02-28T12:03:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34752025-02-28T12:03:40.463ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34762025-02-28T12:03:40.463ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34772025-02-28T12:03:40.464ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34782025-02-28T12:03:40.465ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
34792025-02-28T12:03:40.465ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
34802025-02-28T12:03:40.465ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
34812025-02-28T12:03:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
34822025-02-28T12:03:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
34832025-02-28T12:03:40.467ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
34842025-02-28T12:03:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
34852025-02-28T12:03:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
34862025-02-28T12:03:40.467ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
34872025-02-28T12:03:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
34882025-02-28T12:03:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
34892025-02-28T12:03:40.467ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
34902025-02-28T12:03:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
34912025-02-28T12:03:40.468ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34922025-02-28T12:03:40.470ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
34932025-02-28T12:03:40.470ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34942025-02-28T12:03:40.470ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34952025-02-28T12:03:40.470ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34962025-02-28T12:03:40.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34972025-02-28T12:03:40.470ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
34982025-02-28T12:03:40.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
34992025-02-28T12:03:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
35002025-02-28T12:03:40.473ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35012025-02-28T12:03:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35022025-02-28T12:03:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
35032025-02-28T12:03:40.473ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
35042025-02-28T12:03:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
35052025-02-28T12:03:40.475ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
35062025-02-28T12:03:40.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
35072025-02-28T12:03:40.475ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35082025-02-28T12:03:40.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35092025-02-28T12:03:40.477ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
35102025-02-28T12:03:40.477ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
35112025-02-28T12:03:40.477ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
35122025-02-28T12:03:40.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
35132025-02-28T12:03:40.478ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35142025-02-28T12:03:40.478ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
35152025-02-28T12:03:40.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
35162025-02-28T12:03:40.478ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
35172025-02-28T12:03:40.479ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
35182025-02-28T12:03:40.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
35192025-02-28T12:03:40.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35202025-02-28T12:03:40.479ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
35212025-02-28T12:03:40.480ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1055, tv_nsec: 115009288 } })
35222025-02-28T12:03:40.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
35232025-02-28T12:03:40.480ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
35242025-02-28T12:03:40.480ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35252025-02-28T12:03:40.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35262025-02-28T12:03:40.480ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
35272025-02-28T12:03:40.480ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
35282025-02-28T12:03:40.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
35292025-02-28T12:03:40.481ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
35302025-02-28T12:03:40.481ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 21.214872ms iteration = 10 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
35312025-02-28T12:03:40.483ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35322025-02-28T12:03:40.485ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
35332025-02-28T12:03:40.485ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
35342025-02-28T12:03:40.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
35352025-02-28T12:03:40.511ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 50.048388ms iteration = 10 status = {"errors":[],"garbage_collect_requested":[]}
35362025-02-28T12:03:40.516ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
35372025-02-28T12:03:40.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
35382025-02-28T12:03:40.518ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 56.524844ms iteration = 10 status = {"errors":[],"finish_invoked_ok":[]}
35392025-02-28T12:03:40.519ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 57.402026ms iteration = 10 status = {"errors":[],"requests_created_ok":[]}
35402025-02-28T12:03:40.522ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 61.933845ms iteration = 28 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
35412025-02-28T12:03:40.523ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35422025-02-28T12:03:40.525ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
35432025-02-28T12:03:40.525ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35442025-02-28T12:03:40.526ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 64.516338ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
35452025-02-28T12:03:40.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35462025-02-28T12:03:40.526ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 66.316471ms iteration = 11 status = {"physical_disks_added":0}
35472025-02-28T12:03:40.526ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
35482025-02-28T12:03:40.526ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
35492025-02-28T12:03:40.526ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 66.606558ms iteration = 10 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
35502025-02-28T12:03:40.528ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 68.306679ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
35512025-02-28T12:03:40.529ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 67.792487ms iteration = 10 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
35522025-02-28T12:03:40.590ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
35532025-02-28T12:03:40.591ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35542025-02-28T12:03:40.643ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
35552025-02-28T12:03:40.643ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
35562025-02-28T12:03:40.644ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "169d8038-c996-4561-a378-5335466261c4", "content-length": "26", "date": "Fri, 28 Feb 2025 12:03:40 GMT"} })
35572025-02-28T12:03:40.644ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
35582025-02-28T12:03:40.645ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
35592025-02-28T12:03:40.645ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
35602025-02-28T12:03:40.645ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8af6e00-1a77-4e6a-ab63-f8ea1e49d769", "content-length": "26", "date": "Fri, 28 Feb 2025 12:03:40 GMT"} })
35612025-02-28T12:03:40.645ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
35622025-02-28T12:03:40.646ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
35632025-02-28T12:03:40.647ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
35642025-02-28T12:03:40.689ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
35652025-02-28T12:03:40.689ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
35662025-02-28T12:03:40.689ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "febc8a85-b0bf-4d4f-b506-f35f2d62d744", "content-length": "372", "date": "Fri, 28 Feb 2025 12:03:40 GMT"} })
35672025-02-28T12:03:40.689ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35682025-02-28T12:03:40.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35692025-02-28T12:03:40.691ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35702025-02-28T12:03:40.692ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
35712025-02-28T12:03:40.692ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
35722025-02-28T12:03:40.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
35732025-02-28T12:03:40.702ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
35742025-02-28T12:03:40.702ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 244.610855ms iteration = 28 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
35752025-02-28T12:03:40.725ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35762025-02-28T12:03:40.727ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
35772025-02-28T12:03:40.727ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 267.19471ms iteration = 10 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
35782025-02-28T12:03:40.728ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
35792025-02-28T12:03:40.728ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
35802025-02-28T12:03:40.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35812025-02-28T12:03:40.732ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35822025-02-28T12:03:40.734ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
35832025-02-28T12:03:40.735ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
35842025-02-28T12:03:40.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35852025-02-28T12:03:40.752ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
35862025-02-28T12:03:40.752ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
35872025-02-28T12:03:40.753ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "a0172d2f-32cf-4316-94c6-e3508cbfb499", "date": "Fri, 28 Feb 2025 12:03:40 GMT"} })
35882025-02-28T12:03:40.753ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35892025-02-28T12:03:40.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35902025-02-28T12:03:40.754ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
35912025-02-28T12:03:40.754ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
35922025-02-28T12:03:40.754ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2968352e-b0f6-4ba2-8cc9-41ab112501f7", "content-length": "26", "date": "Fri, 28 Feb 2025 12:03:40 GMT"} })
35932025-02-28T12:03:40.754ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
35942025-02-28T12:03:40.756ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35952025-02-28T12:03:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35962025-02-28T12:03:40.756ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
35972025-02-28T12:03:40.757ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 302.41032ms iteration = 10 status = {}
35982025-02-28T12:03:40.761ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35992025-02-28T12:03:40.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
36002025-02-28T12:03:40.766ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
36012025-02-28T12:03:40.766ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
36022025-02-28T12:03:40.766ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "2e2c8ebf-dd7d-4ca9-9d41-c628aba9e702", "date": "Fri, 28 Feb 2025 12:03:40 GMT"} })
36032025-02-28T12:03:40.767ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 307.568294ms iteration = 10 status = {}
36042025-02-28T12:03:40.800ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
36052025-02-28T12:03:40.800ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "b93afe43-9e34-4cb3-85a6-ae71001e400a", "content-length": "1", "date": "Fri, 28 Feb 2025 12:03:40 GMT"} })
36062025-02-28T12:03:40.803ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
36072025-02-28T12:03:40.803ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
36082025-02-28T12:03:40.805ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 351.040854ms iteration = 10 status = null
36092025-02-28T12:03:40.821ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36102025-02-28T12:03:40.822ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36112025-02-28T12:03:40.824ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36122025-02-28T12:03:40.824ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36132025-02-28T12:03:40.825ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36142025-02-28T12:03:40.828ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
36152025-02-28T12:03:40.828ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36162025-02-28T12:03:40.829ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36172025-02-28T12:03:40.839ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 378.574879ms iteration = 10 status = 0
36182025-02-28T12:03:40.899ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
36192025-02-28T12:03:40.899ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b5abaf0-7bbc-415d-8789-8194bf9fcfc5", "content-length": "691", "date": "Fri, 28 Feb 2025 12:03:40 GMT"} })
36202025-02-28T12:03:40.899ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
36212025-02-28T12:03:40.899ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
36222025-02-28T12:03:40.899ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
36232025-02-28T12:03:40.901ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 440.529765ms iteration = 10 status = {}
36242025-02-28T12:03:40.911ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36252025-02-28T12:03:40.912ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36262025-02-28T12:03:40.914ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
36272025-02-28T12:03:40.914ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36282025-02-28T12:03:40.914ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36292025-02-28T12:03:41.025ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
36302025-02-28T12:03:41.026ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "74568c1f-7899-437a-a8e4-85bf99fb61b4", "content-length": "134", "date": "Fri, 28 Feb 2025 12:03:40 GMT"} })
36312025-02-28T12:03:41.026ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
36322025-02-28T12:03:41.026ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
36332025-02-28T12:03:41.026ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
36342025-02-28T12:03:41.026ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
36352025-02-28T12:03:41.026ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36362025-02-28T12:03:41.026ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36372025-02-28T12:03:41.026ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36382025-02-28T12:03:41.026ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
36392025-02-28T12:03:41.026ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36402025-02-28T12:03:41.026ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
36412025-02-28T12:03:41.026ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
36422025-02-28T12:03:41.026ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36432025-02-28T12:03:41.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36442025-02-28T12:03:41.029ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
36452025-02-28T12:03:41.029ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "b620b358-b035-49da-b560-b22502b90964", "content-length": "2", "date": "Fri, 28 Feb 2025 12:03:40 GMT"} })
36462025-02-28T12:03:41.029ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
36472025-02-28T12:03:41.029ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d104fe2-7e22-4cf7-ac1c-87e52346cb5f", "content-length": "51", "date": "Fri, 28 Feb 2025 12:03:40 GMT"} })
36482025-02-28T12:03:41.029ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
36492025-02-28T12:03:41.029ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
36502025-02-28T12:03:41.196ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "2a897a9c-7bae-448d-a529-379a18703094", "date": "Fri, 28 Feb 2025 12:03:41 GMT"} })
36512025-02-28T12:03:41.196ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
36522025-02-28T12:03:41.196ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
36532025-02-28T12:03:41.196ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7c963ae-7238-489f-9fbc-ac55ed939f99", "content-length": "682", "date": "Fri, 28 Feb 2025 12:03:41 GMT"} })
36542025-02-28T12:03:41.196ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36552025-02-28T12:03:41.196ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36562025-02-28T12:03:41.200ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36572025-02-28T12:03:41.200ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36582025-02-28T12:03:41.203ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36592025-02-28T12:03:41.204ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36602025-02-28T12:03:41.205ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36612025-02-28T12:03:41.206ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36622025-02-28T12:03:41.212ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 751.999684ms iteration = 10 status = {}
36632025-02-28T12:03:44.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 321 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = be476123-63a1-435a-973e-0c1d020830d3 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
36642025-02-28T12:03:50.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 29 reason = Timeout
36652025-02-28T12:03:50.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36662025-02-28T12:03:50.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36672025-02-28T12:03:50.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 29 reason = Timeout
36682025-02-28T12:03:50.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36692025-02-28T12:03:50.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36702025-02-28T12:03:50.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36712025-02-28T12:03:50.462ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36722025-02-28T12:03:50.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36732025-02-28T12:03:50.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
36742025-02-28T12:03:50.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36752025-02-28T12:03:50.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36762025-02-28T12:03:50.463ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.574447ms iteration = 29 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
36772025-02-28T12:03:50.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36782025-02-28T12:03:50.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36792025-02-28T12:03:50.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36802025-02-28T12:03:50.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
36812025-02-28T12:03:50.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36822025-02-28T12:03:50.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36832025-02-28T12:03:50.477ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
36842025-02-28T12:03:50.477ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.383178ms iteration = 29 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
36852025-02-28T12:03:54.255ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = d992a6e3-868c-47ea-83f1-17292dc6fc3d roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
36862025-02-28T12:03:54.255ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = d992a6e3-868c-47ea-83f1-17292dc6fc3d resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
36872025-02-28T12:03:54.258ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2994 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = d992a6e3-868c-47ea-83f1-17292dc6fc3d response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
36882025-02-28T12:03:54.258ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = c59bb617-96cd-4907-9231-3fc795f29838 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
36892025-02-28T12:03:54.258ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = c59bb617-96cd-4907-9231-3fc795f29838 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
36902025-02-28T12:03:54.260ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 1919 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61935 req_id = c59bb617-96cd-4907-9231-3fc795f29838 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
36912025-02-28T12:03:54.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 281 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = da2ae013-9fe4-4c82-8fe4-5e1aebc32436 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
36922025-02-28T12:04:00.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 30 reason = Timeout
36932025-02-28T12:04:00.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36942025-02-28T12:04:00.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36952025-02-28T12:04:00.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 30 reason = Timeout
36962025-02-28T12:04:00.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36972025-02-28T12:04:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36982025-02-28T12:04:00.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36992025-02-28T12:04:00.462ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
37002025-02-28T12:04:00.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
37012025-02-28T12:04:00.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
37022025-02-28T12:04:00.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37032025-02-28T12:04:00.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37042025-02-28T12:04:00.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37052025-02-28T12:04:00.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37062025-02-28T12:04:00.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37072025-02-28T12:04:00.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
37082025-02-28T12:04:00.468ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37092025-02-28T12:04:00.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37102025-02-28T12:04:00.484ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
37112025-02-28T12:04:00.484ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.456258ms iteration = 30 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
37122025-02-28T12:04:00.503ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 43.057569ms iteration = 30 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
37132025-02-28T12:04:04.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 293 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 16f3b358-d8d7-47de-a960-16270efecc10 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
37142025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 6 reason = Timeout
37152025-02-28T12:04:10.455ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-28 11:54:10.455227502 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
37162025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 6 reason = Timeout
37172025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
37182025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
37192025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
37202025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 11 reason = Timeout
37212025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
37222025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 6 reason = Timeout
37232025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 8 reason = Timeout
37242025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
37252025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 11 reason = Timeout
37262025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
37272025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 8 reason = Timeout
37282025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 6 reason = Timeout
37292025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 6 reason = Timeout
37302025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 6 reason = Timeout
37312025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
37322025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
37332025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
37342025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
37352025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
37362025-02-28T12:04:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
37372025-02-28T12:04:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
37382025-02-28T12:04:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
37392025-02-28T12:04:10.456ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5ebff2d9-48b6-4fe6-b852-684bb14b4955.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bddfb0f9-1d7b-45e4-b3a5-35aa6792956b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1085, tv_nsec: 91179383 } })
37402025-02-28T12:04:10.456ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5cefabf1-6831-46ff-98eb-1a3b5abad8b9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1085, tv_nsec: 91362008 } })
37412025-02-28T12:04:10.456ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
37422025-02-28T12:04:10.456ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.129568ms iteration = 6 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
37432025-02-28T12:04:10.456ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
37442025-02-28T12:04:10.456ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.318612ms iteration = 6 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
37452025-02-28T12:04:10.457ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
37462025-02-28T12:04:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
37472025-02-28T12:04:10.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 31 reason = Timeout
37482025-02-28T12:04:10.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37492025-02-28T12:04:10.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 7 reason = Timeout
37502025-02-28T12:04:10.457ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
37512025-02-28T12:04:10.457ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 7.11\u{b5}s iteration = 7 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
37522025-02-28T12:04:10.457ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 11 reason = Timeout
37532025-02-28T12:04:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37542025-02-28T12:04:10.458ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-28 11:54:10.455227502 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
37552025-02-28T12:04:10.458ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.31522ms iteration = 6 status = {"expiration":"2025-02-28T11:54:10.455227502Z","pruned":[]}
37562025-02-28T12:04:10.458ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37572025-02-28T12:04:10.458ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
37582025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
37592025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 31 reason = Timeout
37602025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 6 reason = Timeout
37612025-02-28T12:04:10.459ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
37622025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
37632025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
37642025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 11 reason = Timeout
37652025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
37662025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 11 reason = Timeout
37672025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
37682025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 11 reason = Timeout
37692025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37702025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 11 reason = Timeout
37712025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 11 reason = Timeout
37722025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 2 reason = Timeout
37732025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 6 reason = Timeout
37742025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
37752025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
37762025-02-28T12:04:10.459ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
37772025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
37782025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
37792025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 11 reason = Timeout
37802025-02-28T12:04:10.459ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
37812025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 3 reason = Timeout
37822025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37832025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
37842025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 11 reason = Timeout
37852025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
37862025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
37872025-02-28T12:04:10.459ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 11 reason = Timeout
37882025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 12 reason = Timeout
37892025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
37902025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
37912025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
37922025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
37932025-02-28T12:04:10.460ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1085, tv_nsec: 95279572 } })
37942025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37952025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
37962025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
37972025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
37982025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
37992025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
38002025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38012025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
38022025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
38032025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
38042025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
38052025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
38062025-02-28T12:04:10.460ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
38072025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 6 reason = Timeout
38082025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
38092025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 11 reason = Timeout
38102025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
38112025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 11 reason = Timeout
38122025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
38132025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 11 reason = Timeout
38142025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 11 reason = Timeout
38152025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 6 reason = Timeout
38162025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
38172025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
38182025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
38192025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 11 reason = Timeout
38202025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
38212025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 2 reason = Timeout
38222025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
38232025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
38242025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
38252025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
38262025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
38272025-02-28T12:04:10.460ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1085, tv_nsec: 96004460 } })
38282025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38292025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
38302025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
38312025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
38322025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
38332025-02-28T12:04:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
38342025-02-28T12:04:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
38352025-02-28T12:04:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
38362025-02-28T12:04:10.461ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
38372025-02-28T12:04:10.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38382025-02-28T12:04:10.463ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38392025-02-28T12:04:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
38402025-02-28T12:04:10.463ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
38412025-02-28T12:04:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
38422025-02-28T12:04:10.464ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 6.923907ms iteration = 11 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
38432025-02-28T12:04:10.465ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38442025-02-28T12:04:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
38452025-02-28T12:04:10.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38462025-02-28T12:04:10.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38472025-02-28T12:04:10.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
38482025-02-28T12:04:10.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
38492025-02-28T12:04:10.467ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
38502025-02-28T12:04:10.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
38512025-02-28T12:04:10.468ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38522025-02-28T12:04:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
38532025-02-28T12:04:10.469ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
38542025-02-28T12:04:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
38552025-02-28T12:04:10.470ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38562025-02-28T12:04:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
38572025-02-28T12:04:10.471ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
38582025-02-28T12:04:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
38592025-02-28T12:04:10.473ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38602025-02-28T12:04:10.473ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
38612025-02-28T12:04:10.473ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38622025-02-28T12:04:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
38632025-02-28T12:04:10.473ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38642025-02-28T12:04:10.475ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
38652025-02-28T12:04:10.475ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38662025-02-28T12:04:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
38672025-02-28T12:04:10.476ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38682025-02-28T12:04:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38692025-02-28T12:04:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38702025-02-28T12:04:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
38712025-02-28T12:04:10.476ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
38722025-02-28T12:04:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
38732025-02-28T12:04:10.477ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38742025-02-28T12:04:10.478ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38752025-02-28T12:04:10.478ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
38762025-02-28T12:04:10.478ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
38772025-02-28T12:04:10.478ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
38782025-02-28T12:04:10.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
38792025-02-28T12:04:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
38802025-02-28T12:04:10.480ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
38812025-02-28T12:04:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
38822025-02-28T12:04:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
38832025-02-28T12:04:10.480ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
38842025-02-28T12:04:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
38852025-02-28T12:04:10.484ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38862025-02-28T12:04:10.484ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 24.163744ms iteration = 11 status = {"errors":[],"garbage_collect_requested":[]}
38872025-02-28T12:04:10.486ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
38882025-02-28T12:04:10.486ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
38892025-02-28T12:04:10.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
38902025-02-28T12:04:10.486ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 26.09425ms iteration = 11 status = {"errors":[],"finish_invoked_ok":[]}
38912025-02-28T12:04:10.487ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
38922025-02-28T12:04:10.487ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
38932025-02-28T12:04:10.487ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 26.440219ms iteration = 6 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
38942025-02-28T12:04:10.490ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 29.37695ms iteration = 6 status = {"errors":[],"found_port_ok":[]}
38952025-02-28T12:04:10.491ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
38962025-02-28T12:04:10.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
38972025-02-28T12:04:10.492ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38982025-02-28T12:04:10.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38992025-02-28T12:04:10.493ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1085, tv_nsec: 128437873 } })
39002025-02-28T12:04:10.493ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
39012025-02-28T12:04:10.494ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39022025-02-28T12:04:10.494ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
39032025-02-28T12:04:10.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
39042025-02-28T12:04:10.495ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
39052025-02-28T12:04:10.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
39062025-02-28T12:04:10.495ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
39072025-02-28T12:04:10.496ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
39082025-02-28T12:04:10.496ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
39092025-02-28T12:04:10.496ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39102025-02-28T12:04:10.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
39112025-02-28T12:04:10.499ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39122025-02-28T12:04:10.507ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
39132025-02-28T12:04:10.507ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
39142025-02-28T12:04:10.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
39152025-02-28T12:04:10.508ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
39162025-02-28T12:04:10.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
39172025-02-28T12:04:10.512ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
39182025-02-28T12:04:10.513ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
39192025-02-28T12:04:10.517ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
39202025-02-28T12:04:10.517ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
39212025-02-28T12:04:10.517ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
39222025-02-28T12:04:10.517ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
39232025-02-28T12:04:10.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
39242025-02-28T12:04:10.519ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
39252025-02-28T12:04:10.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
39262025-02-28T12:04:10.529ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
39272025-02-28T12:04:10.529ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
39282025-02-28T12:04:10.529ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
39292025-02-28T12:04:10.529ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
39302025-02-28T12:04:10.530ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
39312025-02-28T12:04:10.532ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
39322025-02-28T12:04:10.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
39332025-02-28T12:04:10.535ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
39342025-02-28T12:04:10.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
39352025-02-28T12:04:10.537ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
39362025-02-28T12:04:10.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
39372025-02-28T12:04:10.538ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 78.939536ms iteration = 31 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
39382025-02-28T12:04:10.538ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
39392025-02-28T12:04:10.538ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
39402025-02-28T12:04:10.538ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 79.188322ms iteration = 6 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
39412025-02-28T12:04:10.540ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39422025-02-28T12:04:10.542ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
39432025-02-28T12:04:10.542ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39442025-02-28T12:04:10.543ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
39452025-02-28T12:04:10.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39462025-02-28T12:04:10.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
39472025-02-28T12:04:10.544ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39482025-02-28T12:04:10.545ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
39492025-02-28T12:04:10.545ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 85.977095ms iteration = 11 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
39502025-02-28T12:04:10.547ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
39512025-02-28T12:04:10.547ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
39522025-02-28T12:04:10.548ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
39532025-02-28T12:04:10.549ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
39542025-02-28T12:04:10.549ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
39552025-02-28T12:04:10.549ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
39562025-02-28T12:04:10.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
39572025-02-28T12:04:10.557ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
39582025-02-28T12:04:10.558ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
39592025-02-28T12:04:10.563ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
39602025-02-28T12:04:10.564ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
39612025-02-28T12:04:10.565ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
39622025-02-28T12:04:10.566ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
39632025-02-28T12:04:10.576ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39642025-02-28T12:04:10.576ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39652025-02-28T12:04:10.646ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
39662025-02-28T12:04:10.646ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
39672025-02-28T12:04:10.647ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3786f66a-30cc-407c-81c9-2cba52998100", "content-length": "26", "date": "Fri, 28 Feb 2025 12:04:10 GMT"} })
39682025-02-28T12:04:10.647ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
39692025-02-28T12:04:10.649ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
39702025-02-28T12:04:10.657ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 196.815081ms iteration = 11 status = {"errors":[],"requests_created_ok":[]}
39712025-02-28T12:04:10.708ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
39722025-02-28T12:04:10.709ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
39732025-02-28T12:04:10.709ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "58f14b91-b039-44f5-8499-13c3cde88ba0", "content-length": "372", "date": "Fri, 28 Feb 2025 12:04:10 GMT"} })
39742025-02-28T12:04:10.709ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39752025-02-28T12:04:10.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
39762025-02-28T12:04:10.710ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39772025-02-28T12:04:10.712ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
39782025-02-28T12:04:10.713ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
39792025-02-28T12:04:10.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
39802025-02-28T12:04:10.715ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39812025-02-28T12:04:10.718ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
39822025-02-28T12:04:10.718ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
39832025-02-28T12:04:10.718ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
39842025-02-28T12:04:10.720ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39852025-02-28T12:04:10.721ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
39862025-02-28T12:04:10.722ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
39872025-02-28T12:04:10.722ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
39882025-02-28T12:04:10.722ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
39892025-02-28T12:04:10.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
39902025-02-28T12:04:10.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
39912025-02-28T12:04:10.725ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39922025-02-28T12:04:10.727ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
39932025-02-28T12:04:10.727ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
39942025-02-28T12:04:10.727ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
39952025-02-28T12:04:10.730ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39962025-02-28T12:04:10.734ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
39972025-02-28T12:04:10.734ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
39982025-02-28T12:04:10.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
39992025-02-28T12:04:10.734ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
40002025-02-28T12:04:10.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
40012025-02-28T12:04:10.737ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40022025-02-28T12:04:10.737ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
40032025-02-28T12:04:10.737ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 279.473136ms iteration = 31 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
40042025-02-28T12:04:10.737ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 277.365525ms iteration = 11 status = 0
40052025-02-28T12:04:10.739ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
40062025-02-28T12:04:10.739ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
40072025-02-28T12:04:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
40082025-02-28T12:04:10.744ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40092025-02-28T12:04:10.747ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
40102025-02-28T12:04:10.747ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
40112025-02-28T12:04:10.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
40122025-02-28T12:04:10.753ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 293.952066ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
40132025-02-28T12:04:10.759ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
40142025-02-28T12:04:10.759ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
40152025-02-28T12:04:10.761ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
40162025-02-28T12:04:10.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
40172025-02-28T12:04:10.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
40182025-02-28T12:04:10.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
40192025-02-28T12:04:10.767ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40202025-02-28T12:04:10.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
40212025-02-28T12:04:10.769ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
40222025-02-28T12:04:10.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
40232025-02-28T12:04:10.777ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40242025-02-28T12:04:10.778ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
40252025-02-28T12:04:10.778ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
40262025-02-28T12:04:10.779ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "566f9518-13a9-40ba-8829-10c58b1b3ea3", "content-length": "26", "date": "Fri, 28 Feb 2025 12:04:10 GMT"} })
40272025-02-28T12:04:10.779ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
40282025-02-28T12:04:10.779ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
40292025-02-28T12:04:10.779ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
40302025-02-28T12:04:10.779ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
40312025-02-28T12:04:10.779ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
40322025-02-28T12:04:10.779ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 319.79954ms iteration = 11 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
40332025-02-28T12:04:10.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
40342025-02-28T12:04:10.780ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
40352025-02-28T12:04:10.781ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
40362025-02-28T12:04:10.782ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
40372025-02-28T12:04:10.782ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 322.28946ms iteration = 11 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
40382025-02-28T12:04:10.786ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
40392025-02-28T12:04:10.786ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 0 version = 2
40402025-02-28T12:04:10.786ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
40412025-02-28T12:04:10.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
40422025-02-28T12:04:10.787ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 327.160697ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
40432025-02-28T12:04:10.793ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-28T11:59:08.719399Z, time_modified: 2025-02-28T11:59:08.719399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }) }
40442025-02-28T12:04:10.793ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 338.184014ms iteration = 6 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]}},"default_endpoint":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
40452025-02-28T12:04:10.806ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
40462025-02-28T12:04:10.806ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
40472025-02-28T12:04:10.807ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "ee775030-52e7-4ff2-b907-3e4b87ce1740", "date": "Fri, 28 Feb 2025 12:04:10 GMT"} })
40482025-02-28T12:04:10.807ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40492025-02-28T12:04:10.807ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40502025-02-28T12:04:10.848ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 388.475358ms iteration = 12 status = {"physical_disks_added":0}
40512025-02-28T12:04:10.852ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
40522025-02-28T12:04:10.852ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
40532025-02-28T12:04:10.853ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e41192d1-61e3-4320-9fac-7721674a48a3", "content-length": "26", "date": "Fri, 28 Feb 2025 12:04:10 GMT"} })
40542025-02-28T12:04:10.853ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
40552025-02-28T12:04:10.854ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
40562025-02-28T12:04:10.858ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
40572025-02-28T12:04:10.858ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 398.647614ms iteration = 6 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
40582025-02-28T12:04:10.943ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
40592025-02-28T12:04:10.944ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc765e17-205c-44fc-9cdc-d074f3193d50", "content-length": "691", "date": "Fri, 28 Feb 2025 12:04:10 GMT"} })
40602025-02-28T12:04:10.944ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
40612025-02-28T12:04:10.944ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
40622025-02-28T12:04:10.944ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
40632025-02-28T12:04:10.947ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 487.55531ms iteration = 11 status = {}
40642025-02-28T12:04:10.951ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40652025-02-28T12:04:10.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
40662025-02-28T12:04:10.955ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
40672025-02-28T12:04:10.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
40682025-02-28T12:04:11.007ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 546.846731ms iteration = 11 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
40692025-02-28T12:04:11.020ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 564.99427ms iteration = 11 status = {}
40702025-02-28T12:04:11.022ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
40712025-02-28T12:04:11.071ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
40722025-02-28T12:04:11.071ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "20d4d625-d988-4d76-b6d5-84277ae77755", "content-length": "1", "date": "Fri, 28 Feb 2025 12:04:10 GMT"} })
40732025-02-28T12:04:11.074ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
40742025-02-28T12:04:11.074ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
40752025-02-28T12:04:11.134ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
40762025-02-28T12:04:11.135ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "82b6d6e7-23a8-4568-a4f6-164bfa043480", "date": "Fri, 28 Feb 2025 12:04:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
40772025-02-28T12:04:11.135ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
40782025-02-28T12:04:11.137ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "8d8230e9-8659-4ca8-8daf-11bb240db262", "date": "Fri, 28 Feb 2025 12:04:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
40792025-02-28T12:04:11.137ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
40802025-02-28T12:04:11.142ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
40812025-02-28T12:04:11.142ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 686.844011ms iteration = 8 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
40822025-02-28T12:04:11.143ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
40832025-02-28T12:04:11.143ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
40842025-02-28T12:04:11.146ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
40852025-02-28T12:04:11.146ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "4098ab7c-005e-4c27-921d-f6e755aae856", "date": "Fri, 28 Feb 2025 12:04:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
40862025-02-28T12:04:11.146ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
40872025-02-28T12:04:11.148ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "91a72a5a-442b-4069-9884-abfc4c95d850", "date": "Fri, 28 Feb 2025 12:04:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
40882025-02-28T12:04:11.148ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
40892025-02-28T12:04:11.148ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40902025-02-28T12:04:11.149ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40912025-02-28T12:04:11.149ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 694.037755ms iteration = 11 status = null
40922025-02-28T12:04:11.150ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "50c47da1-4539-402a-b89c-20ed303f62b2", "date": "Fri, 28 Feb 2025 12:04:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
40932025-02-28T12:04:11.150ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
40942025-02-28T12:04:11.151ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40952025-02-28T12:04:11.152ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
40962025-02-28T12:04:11.152ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 696.76764ms iteration = 8 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
40972025-02-28T12:04:11.152ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40982025-02-28T12:04:11.155ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
40992025-02-28T12:04:11.155ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
41002025-02-28T12:04:11.160ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "c03c6321-3626-4f8e-805c-1661dd8723eb", "date": "Fri, 28 Feb 2025 12:04:10 GMT"} })
41012025-02-28T12:04:11.161ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 702.014048ms iteration = 11 status = {}
41022025-02-28T12:04:11.190ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
41032025-02-28T12:04:11.190ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
41042025-02-28T12:04:11.190ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
41052025-02-28T12:04:11.190ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
41062025-02-28T12:04:11.193ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41072025-02-28T12:04:11.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41082025-02-28T12:04:11.196ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41092025-02-28T12:04:11.199ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
41102025-02-28T12:04:11.199ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
41112025-02-28T12:04:11.199ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
41122025-02-28T12:04:11.201ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41132025-02-28T12:04:11.202ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41142025-02-28T12:04:11.203ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
41152025-02-28T12:04:11.203ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
41162025-02-28T12:04:11.203ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
41172025-02-28T12:04:11.205ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 1 version = 2
41182025-02-28T12:04:11.205ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
41192025-02-28T12:04:11.205ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-02-28 11:59:52.542241 UTC dns_group = external generation = 2
41202025-02-28T12:04:11.205ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 750.394655ms iteration = 6 status = {"generation":2}
41212025-02-28T12:04:11.207ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
41222025-02-28T12:04:11.207ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
41232025-02-28T12:04:11.207ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
41242025-02-28T12:04:11.211ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41252025-02-28T12:04:11.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41262025-02-28T12:04:11.213ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
41272025-02-28T12:04:11.213ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41282025-02-28T12:04:11.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41292025-02-28T12:04:11.222ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41302025-02-28T12:04:11.227ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
41312025-02-28T12:04:11.227ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
41322025-02-28T12:04:11.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
41332025-02-28T12:04:11.229ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
41342025-02-28T12:04:11.229ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "508b8b16-c3cc-47f2-8284-0f1a1252c32e", "content-length": "134", "date": "Fri, 28 Feb 2025 12:04:10 GMT"} })
41352025-02-28T12:04:11.229ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
41362025-02-28T12:04:11.229ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
41372025-02-28T12:04:11.230ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
41382025-02-28T12:04:11.230ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
41392025-02-28T12:04:11.230ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
41402025-02-28T12:04:11.230ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
41412025-02-28T12:04:11.230ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
41422025-02-28T12:04:11.230ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
41432025-02-28T12:04:11.230ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
41442025-02-28T12:04:11.230ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
41452025-02-28T12:04:11.230ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
41462025-02-28T12:04:11.230ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
41472025-02-28T12:04:11.230ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-02-28 11:59:51.915187 UTC dns_group = internal generation = 1
41482025-02-28T12:04:11.230ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 775.347548ms iteration = 6 status = {"generation":1}
41492025-02-28T12:04:11.230ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
41502025-02-28T12:04:11.230ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41512025-02-28T12:04:11.230ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41522025-02-28T12:04:11.233ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
41532025-02-28T12:04:11.233ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac932ea3-c9b8-4bad-91f0-40c8ca881e60", "content-length": "2", "date": "Fri, 28 Feb 2025 12:04:10 GMT"} })
41542025-02-28T12:04:11.233ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
41552025-02-28T12:04:11.233ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c43afa8-0058-403b-bee2-255019aefc34", "content-length": "51", "date": "Fri, 28 Feb 2025 12:04:10 GMT"} })
41562025-02-28T12:04:11.233ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
41572025-02-28T12:04:11.233ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
41582025-02-28T12:04:11.238ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41592025-02-28T12:04:11.238ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41602025-02-28T12:04:11.241ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41612025-02-28T12:04:11.246ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
41622025-02-28T12:04:11.246ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41632025-02-28T12:04:11.247ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41642025-02-28T12:04:11.248ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
41652025-02-28T12:04:11.249ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41662025-02-28T12:04:11.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41672025-02-28T12:04:11.251ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41682025-02-28T12:04:11.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41692025-02-28T12:04:11.253ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41702025-02-28T12:04:11.253ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41712025-02-28T12:04:11.255ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41722025-02-28T12:04:11.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41732025-02-28T12:04:11.257ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41742025-02-28T12:04:11.257ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41752025-02-28T12:04:11.258ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41762025-02-28T12:04:11.258ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41772025-02-28T12:04:11.291ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41782025-02-28T12:04:11.295ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
41792025-02-28T12:04:11.295ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41802025-02-28T12:04:11.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41812025-02-28T12:04:11.297ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41822025-02-28T12:04:11.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41832025-02-28T12:04:11.299ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41842025-02-28T12:04:11.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41852025-02-28T12:04:11.302ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41862025-02-28T12:04:11.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41872025-02-28T12:04:11.304ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41882025-02-28T12:04:11.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41892025-02-28T12:04:11.306ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41902025-02-28T12:04:11.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41912025-02-28T12:04:11.308ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41922025-02-28T12:04:11.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41932025-02-28T12:04:11.309ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41942025-02-28T12:04:11.313ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
41952025-02-28T12:04:11.313ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41962025-02-28T12:04:11.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41972025-02-28T12:04:11.314ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41982025-02-28T12:04:11.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
41992025-02-28T12:04:11.316ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42002025-02-28T12:04:11.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42012025-02-28T12:04:11.317ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42022025-02-28T12:04:11.324ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
42032025-02-28T12:04:11.324ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42042025-02-28T12:04:11.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42052025-02-28T12:04:11.334ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42062025-02-28T12:04:11.335ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42072025-02-28T12:04:11.336ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42082025-02-28T12:04:11.336ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42092025-02-28T12:04:11.338ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42102025-02-28T12:04:11.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42112025-02-28T12:04:11.340ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42122025-02-28T12:04:11.340ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42132025-02-28T12:04:11.342ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42142025-02-28T12:04:11.342ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42152025-02-28T12:04:11.344ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42162025-02-28T12:04:11.344ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42172025-02-28T12:04:11.345ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42182025-02-28T12:04:11.345ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42192025-02-28T12:04:11.347ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42202025-02-28T12:04:11.350ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
42212025-02-28T12:04:11.350ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42222025-02-28T12:04:11.350ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42232025-02-28T12:04:11.351ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42242025-02-28T12:04:11.352ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42252025-02-28T12:04:11.353ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42262025-02-28T12:04:11.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42272025-02-28T12:04:11.354ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42282025-02-28T12:04:11.355ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "77c5f5b0-414f-4c6d-b6af-3d165c64a997", "date": "Fri, 28 Feb 2025 12:04:11 GMT"} })
42292025-02-28T12:04:11.355ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
42302025-02-28T12:04:11.355ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
42312025-02-28T12:04:11.356ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "6301d939-842b-42fa-8802-ac126ba72f1e", "content-length": "682", "date": "Fri, 28 Feb 2025 12:04:11 GMT"} })
42322025-02-28T12:04:11.356ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42332025-02-28T12:04:11.356ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42342025-02-28T12:04:11.356ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "49d9e57e-f662-4870-a9b7-9f757419ba92", "content-length": "2", "date": "Fri, 28 Feb 2025 12:04:11 GMT"} })
42352025-02-28T12:04:11.359ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:619 sled = http://[fd00:1122:3344:101::1]:12345
42362025-02-28T12:04:11.359ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42372025-02-28T12:04:11.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42382025-02-28T12:04:11.360ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
42392025-02-28T12:04:11.360ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42402025-02-28T12:04:11.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42412025-02-28T12:04:11.361ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 900.995341ms iteration = 2 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"operation":"list","target_sled":"50ac5f18-4748-43d4-a4ac-8d2114194efc","time":"2025-02-28T12:04:11.359459081Z"}]}
42422025-02-28T12:04:11.367ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42432025-02-28T12:04:11.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42442025-02-28T12:04:11.371ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: efece645-4fc7-4f5a-900b-9b0d58342421, kind: Service { id: db281835-e35c-480d-b1a2-07f321816d55 }, name: Name("external-dns-db281835-e35c-480d-b1a2-07f321816d55"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 88])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fb9b46af-fe90-4b4d-b831-9e9a193e0996, kind: Service { id: 5cefabf1-6831-46ff-98eb-1a3b5abad8b9 }, name: Name("external-dns-5cefabf1-6831-46ff-98eb-1a3b5abad8b9"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 243, 164])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 08731947-b4fa-4741-bf20-8ff14850f21a, kind: Service { id: 3bbfd7d6-3f72-4034-ad35-e956e1c12923 }, name: Name("nexus-3bbfd7d6-3f72-4034-ad35-e956e1c12923"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 4814ec6a-93e1-40fa-ba73-651e41bc6ee3, kind: Service { id: c1b34bc5-1c2f-4d3c-808c-b7a441aac295 }, name: Name("nexus-c1b34bc5-1c2f-4d3c-808c-b7a441aac295"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 152])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e56f48b3-247c-430d-8e79-57098c59bab5, kind: Service { id: e594478d-1d06-4692-a5d7-cded9c00f6d4 }, name: Name("nexus-e594478d-1d06-4692-a5d7-cded9c00f6d4"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 182, 65])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
42452025-02-28T12:04:11.371ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42462025-02-28T12:04:11.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42472025-02-28T12:04:11.375ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42482025-02-28T12:04:11.376ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42492025-02-28T12:04:11.376ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42502025-02-28T12:04:11.377ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
42512025-02-28T12:04:11.377ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42522025-02-28T12:04:11.377ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
42532025-02-28T12:04:11.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42542025-02-28T12:04:11.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
42552025-02-28T12:04:11.379ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42562025-02-28T12:04:11.379ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
42572025-02-28T12:04:11.379ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: efece645-4fc7-4f5a-900b-9b0d58342421, kind: Service { id: db281835-e35c-480d-b1a2-07f321816d55 }, name: Name("external-dns-db281835-e35c-480d-b1a2-07f321816d55"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 88])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fb9b46af-fe90-4b4d-b831-9e9a193e0996, kind: Service { id: 5cefabf1-6831-46ff-98eb-1a3b5abad8b9 }, name: Name("external-dns-5cefabf1-6831-46ff-98eb-1a3b5abad8b9"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 243, 164])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 08731947-b4fa-4741-bf20-8ff14850f21a, kind: Service { id: 3bbfd7d6-3f72-4034-ad35-e956e1c12923 }, name: Name("nexus-3bbfd7d6-3f72-4034-ad35-e956e1c12923"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 4814ec6a-93e1-40fa-ba73-651e41bc6ee3, kind: Service { id: c1b34bc5-1c2f-4d3c-808c-b7a441aac295 }, name: Name("nexus-c1b34bc5-1c2f-4d3c-808c-b7a441aac295"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 152])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e56f48b3-247c-430d-8e79-57098c59bab5, kind: Service { id: e594478d-1d06-4692-a5d7-cded9c00f6d4 }, name: Name("nexus-e594478d-1d06-4692-a5d7-cded9c00f6d4"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 182, 65])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
42582025-02-28T12:04:11.379ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
42592025-02-28T12:04:11.380ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
42602025-02-28T12:04:11.380ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
42612025-02-28T12:04:11.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
42622025-02-28T12:04:11.383ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 923.841252ms iteration = 11 status = {}
42632025-02-28T12:04:11.384ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
42642025-02-28T12:04:11.384ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
42652025-02-28T12:04:11.386ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42662025-02-28T12:04:11.388ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
42672025-02-28T12:04:11.388ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
42682025-02-28T12:04:11.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
42692025-02-28T12:04:11.390ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
42702025-02-28T12:04:11.390ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
42712025-02-28T12:04:11.392ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42722025-02-28T12:04:11.394ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
42732025-02-28T12:04:11.394ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
42742025-02-28T12:04:11.394ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
42752025-02-28T12:04:11.398ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
42762025-02-28T12:04:11.398ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
42772025-02-28T12:04:11.404ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42782025-02-28T12:04:11.406ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
42792025-02-28T12:04:11.406ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
42802025-02-28T12:04:11.406ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
42812025-02-28T12:04:11.413ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
42822025-02-28T12:04:11.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
42832025-02-28T12:04:11.415ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42842025-02-28T12:04:11.416ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
42852025-02-28T12:04:11.416ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
42862025-02-28T12:04:11.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
42872025-02-28T12:04:11.428ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
42882025-02-28T12:04:11.428ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
42892025-02-28T12:04:11.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42902025-02-28T12:04:11.463ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
42912025-02-28T12:04:11.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
42922025-02-28T12:04:11.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
42932025-02-28T12:04:11.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
42942025-02-28T12:04:11.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
42952025-02-28T12:04:11.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42962025-02-28T12:04:11.468ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
42972025-02-28T12:04:11.468ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
42982025-02-28T12:04:11.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
42992025-02-28T12:04:11.476ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
43002025-02-28T12:04:11.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
43012025-02-28T12:04:11.477ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 50ac5f18-4748-43d4-a4ac-8d2114194efc, time_created: 2025-02-28T11:59:46.871334Z, time_modified: 2025-02-28T11:59:46.871334Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
43022025-02-28T12:04:11.477ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
43032025-02-28T12:04:11.477ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
43042025-02-28T12:04:11.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
43052025-02-28T12:04:11.477ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43062025-02-28T12:04:11.479ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
43072025-02-28T12:04:11.480ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
43082025-02-28T12:04:11.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
43092025-02-28T12:04:11.480ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
43102025-02-28T12:04:11.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
43112025-02-28T12:04:11.481ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
43122025-02-28T12:04:11.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
43132025-02-28T12:04:11.483ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43142025-02-28T12:04:11.484ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43152025-02-28T12:04:11.485ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
43162025-02-28T12:04:11.485ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
43172025-02-28T12:04:11.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
43182025-02-28T12:04:11.486ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
43192025-02-28T12:04:11.486ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43202025-02-28T12:04:11.488ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
43212025-02-28T12:04:11.488ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
43222025-02-28T12:04:11.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
43232025-02-28T12:04:11.497ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
43242025-02-28T12:04:11.499ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 1.03764245s iteration = 3 status = {"blueprint_id":"7a676565-49af-418f-9d1f-e6c070e71120","inventory_collection_id":"15665a6a-6930-47e8-ae90-d9ea47eeca82","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":8,"num_already_tombstoned":0,"num_inserted":1,"num_not_in_inventory":0,"num_tombstoned":0}}}
43252025-02-28T12:04:11.519ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
43262025-02-28T12:04:11.523ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "1c4c6da6-5168-4be6-b49b-0e8c54566082", "date": "Fri, 28 Feb 2025 12:04:11 GMT"} })
43272025-02-28T12:04:11.525ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
43282025-02-28T12:04:11.526ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 1.066174809s
43292025-02-28T12:04:11.526ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 1.066183399s iteration = 2 status = {}
43302025-02-28T12:04:14.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 275 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 30afe961-c32d-4fe3-b96e-1dfba2e2de7b response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
43312025-02-28T12:04:20.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 32 reason = Timeout
43322025-02-28T12:04:20.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43332025-02-28T12:04:20.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43342025-02-28T12:04:20.460ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 32 reason = Timeout
43352025-02-28T12:04:20.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43362025-02-28T12:04:20.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43372025-02-28T12:04:20.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43382025-02-28T12:04:20.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
43392025-02-28T12:04:20.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43402025-02-28T12:04:20.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43412025-02-28T12:04:20.469ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43422025-02-28T12:04:20.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43432025-02-28T12:04:20.471ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43442025-02-28T12:04:20.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
43452025-02-28T12:04:20.473ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43462025-02-28T12:04:20.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43472025-02-28T12:04:20.490ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
43482025-02-28T12:04:20.490ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 32.195257ms iteration = 32 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
43492025-02-28T12:04:20.492ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43502025-02-28T12:04:20.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43512025-02-28T12:04:20.524ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.232381ms iteration = 32 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
43522025-02-28T12:04:24.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 253 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 72ffd5a3-1474-4bb5-880d-e60ddc096c65 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
43532025-02-28T12:04:30.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 33 reason = Timeout
43542025-02-28T12:04:30.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43552025-02-28T12:04:30.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43562025-02-28T12:04:30.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 33 reason = Timeout
43572025-02-28T12:04:30.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43582025-02-28T12:04:30.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43592025-02-28T12:04:30.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43602025-02-28T12:04:30.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
43612025-02-28T12:04:30.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43622025-02-28T12:04:30.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43632025-02-28T12:04:30.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43642025-02-28T12:04:30.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43652025-02-28T12:04:30.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43662025-02-28T12:04:30.470ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
43672025-02-28T12:04:30.470ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43682025-02-28T12:04:30.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43692025-02-28T12:04:30.482ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
43702025-02-28T12:04:30.482ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.295887ms iteration = 33 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
43712025-02-28T12:04:30.488ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43722025-02-28T12:04:30.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43732025-02-28T12:04:30.490ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.225424ms iteration = 33 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
43742025-02-28T12:04:34.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 421 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 502a1973-cfdb-4336-a126-4060db1a88ca response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
43752025-02-28T12:04:40.455ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 12 reason = Timeout
43762025-02-28T12:04:40.455ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
43772025-02-28T12:04:40.455ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 12 reason = Timeout
43782025-02-28T12:04:40.455ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
43792025-02-28T12:04:40.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
43802025-02-28T12:04:40.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
43812025-02-28T12:04:40.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 34 reason = Timeout
43822025-02-28T12:04:40.457ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 12 reason = Timeout
43832025-02-28T12:04:40.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43842025-02-28T12:04:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43852025-02-28T12:04:40.457ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1115, tv_nsec: 93142411 } })
43862025-02-28T12:04:40.458ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
43872025-02-28T12:04:40.458ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1115, tv_nsec: 93806547 } })
43882025-02-28T12:04:40.458ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
43892025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43902025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 34 reason = Timeout
43912025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43922025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 12 reason = Timeout
43932025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
43942025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 12 reason = Timeout
43952025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
43962025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
43972025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 13 reason = Timeout
43982025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 12 reason = Timeout
43992025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
44002025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
44012025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 12 reason = Timeout
44022025-02-28T12:04:40.459ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
44032025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 12 reason = Timeout
44042025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
44052025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
44062025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 12 reason = Timeout
44072025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
44082025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 12 reason = Timeout
44092025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
44102025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 12 reason = Timeout
44112025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44122025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44132025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
44142025-02-28T12:04:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
44152025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
44162025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
44172025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
44182025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
44192025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44202025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
44212025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
44222025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 12 reason = Timeout
44232025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
44242025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 12 reason = Timeout
44252025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
44262025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 12 reason = Timeout
44272025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 12 reason = Timeout
44282025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
44292025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
44302025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 12 reason = Timeout
44312025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
44322025-02-28T12:04:40.460ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.31914ms iteration = 12 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
44332025-02-28T12:04:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
44342025-02-28T12:04:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
44352025-02-28T12:04:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
44362025-02-28T12:04:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
44372025-02-28T12:04:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
44382025-02-28T12:04:40.461ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44392025-02-28T12:04:40.461ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44402025-02-28T12:04:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44412025-02-28T12:04:40.462ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44422025-02-28T12:04:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
44432025-02-28T12:04:40.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44442025-02-28T12:04:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44452025-02-28T12:04:40.464ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44462025-02-28T12:04:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
44472025-02-28T12:04:40.464ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
44482025-02-28T12:04:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
44492025-02-28T12:04:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
44502025-02-28T12:04:40.464ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
44512025-02-28T12:04:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
44522025-02-28T12:04:40.465ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
44532025-02-28T12:04:40.465ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
44542025-02-28T12:04:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
44552025-02-28T12:04:40.466ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44562025-02-28T12:04:40.467ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44572025-02-28T12:04:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
44582025-02-28T12:04:40.468ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
44592025-02-28T12:04:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
44602025-02-28T12:04:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
44612025-02-28T12:04:40.469ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
44622025-02-28T12:04:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
44632025-02-28T12:04:40.472ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44642025-02-28T12:04:40.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44652025-02-28T12:04:40.473ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
44662025-02-28T12:04:40.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
44672025-02-28T12:04:40.474ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
44682025-02-28T12:04:40.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
44692025-02-28T12:04:40.475ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
44702025-02-28T12:04:40.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
44712025-02-28T12:04:40.501ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
44722025-02-28T12:04:40.501ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
44732025-02-28T12:04:40.502ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f23b204-d3e1-4183-92b6-c66c0a4baf15", "content-length": "26", "date": "Fri, 28 Feb 2025 12:04:40 GMT"} })
44742025-02-28T12:04:40.502ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
44752025-02-28T12:04:40.504ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
44762025-02-28T12:04:40.504ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
44772025-02-28T12:04:40.504ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
44782025-02-28T12:04:40.504ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebcf2541-8d04-4370-9d6d-8e6612b09a15", "content-length": "26", "date": "Fri, 28 Feb 2025 12:04:40 GMT"} })
44792025-02-28T12:04:40.504ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
44802025-02-28T12:04:40.506ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
44812025-02-28T12:04:40.506ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
44822025-02-28T12:04:40.507ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
44832025-02-28T12:04:40.514ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
44842025-02-28T12:04:40.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
44852025-02-28T12:04:40.518ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 57.937689ms iteration = 12 status = {"errors":[],"requests_created_ok":[]}
44862025-02-28T12:04:40.519ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 58.277667ms iteration = 12 status = {"errors":[],"garbage_collect_requested":[]}
44872025-02-28T12:04:40.519ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
44882025-02-28T12:04:40.519ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
44892025-02-28T12:04:40.519ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
44902025-02-28T12:04:40.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
44912025-02-28T12:04:40.520ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
44922025-02-28T12:04:40.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
44932025-02-28T12:04:40.522ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44942025-02-28T12:04:40.522ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
44952025-02-28T12:04:40.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
44962025-02-28T12:04:40.523ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44972025-02-28T12:04:40.524ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
44982025-02-28T12:04:40.524ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
44992025-02-28T12:04:40.524ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 65.408999ms iteration = 34 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
45002025-02-28T12:04:40.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
45012025-02-28T12:04:40.525ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
45022025-02-28T12:04:40.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
45032025-02-28T12:04:40.526ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
45042025-02-28T12:04:40.526ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
45052025-02-28T12:04:40.526ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 65.788058ms iteration = 12 status = {"errors":[],"finish_invoked_ok":[]}
45062025-02-28T12:04:40.526ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45072025-02-28T12:04:40.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
45082025-02-28T12:04:40.528ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45092025-02-28T12:04:40.528ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
45102025-02-28T12:04:40.529ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1115, tv_nsec: 164570775 } })
45112025-02-28T12:04:40.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
45122025-02-28T12:04:40.529ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
45132025-02-28T12:04:40.529ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45142025-02-28T12:04:40.529ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
45152025-02-28T12:04:40.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45162025-02-28T12:04:40.530ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
45172025-02-28T12:04:40.530ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 70.631885ms iteration = 12 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
45182025-02-28T12:04:40.531ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
45192025-02-28T12:04:40.531ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45202025-02-28T12:04:40.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45212025-02-28T12:04:40.533ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45222025-02-28T12:04:40.535ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
45232025-02-28T12:04:40.535ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
45242025-02-28T12:04:40.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
45252025-02-28T12:04:40.544ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
45262025-02-28T12:04:40.544ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 86.801746ms iteration = 34 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
45272025-02-28T12:04:40.554ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45282025-02-28T12:04:40.556ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
45292025-02-28T12:04:40.556ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
45302025-02-28T12:04:40.556ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
45312025-02-28T12:04:40.556ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
45322025-02-28T12:04:40.556ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
45332025-02-28T12:04:40.556ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
45342025-02-28T12:04:40.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
45352025-02-28T12:04:40.558ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 99.292787ms iteration = 12 status = 0
45362025-02-28T12:04:40.562ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 101.386707ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
45372025-02-28T12:04:40.562ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
45382025-02-28T12:04:40.562ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 103.096398ms iteration = 12 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
45392025-02-28T12:04:40.565ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 106.204384ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
45402025-02-28T12:04:40.567ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 106.670025ms iteration = 12 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
45412025-02-28T12:04:40.584ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
45422025-02-28T12:04:40.584ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7d83b07-653c-48d2-8b26-e6ff7bf3ce4d", "content-length": "1", "date": "Fri, 28 Feb 2025 12:04:40 GMT"} })
45432025-02-28T12:04:40.586ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
45442025-02-28T12:04:40.586ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
45452025-02-28T12:04:40.591ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 136.075835ms iteration = 12 status = null
45462025-02-28T12:04:40.594ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 139.416715ms iteration = 12 status = {}
45472025-02-28T12:04:40.601ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
45482025-02-28T12:04:40.602ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
45492025-02-28T12:04:40.602ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 142.47147ms iteration = 12 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
45502025-02-28T12:04:40.605ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
45512025-02-28T12:04:40.606ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
45522025-02-28T12:04:40.606ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "aee61a8d-fbe3-4f59-a873-e08605d12561", "content-length": "26", "date": "Fri, 28 Feb 2025 12:04:40 GMT"} })
45532025-02-28T12:04:40.606ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
45542025-02-28T12:04:40.608ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
45552025-02-28T12:04:40.610ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
45562025-02-28T12:04:40.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
45572025-02-28T12:04:40.669ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 209.458916ms iteration = 13 status = {"physical_disks_added":0}
45582025-02-28T12:04:40.679ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
45592025-02-28T12:04:40.679ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
45602025-02-28T12:04:40.680ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "98168ca6-352a-4ea4-80aa-9d417672ba94", "content-length": "372", "date": "Fri, 28 Feb 2025 12:04:40 GMT"} })
45612025-02-28T12:04:40.680ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45622025-02-28T12:04:40.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45632025-02-28T12:04:40.703ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
45642025-02-28T12:04:40.703ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
45652025-02-28T12:04:40.703ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "df052450-c3ab-4685-b230-cf221779945f", "date": "Fri, 28 Feb 2025 12:04:40 GMT"} })
45662025-02-28T12:04:40.703ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45672025-02-28T12:04:40.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45682025-02-28T12:04:40.705ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45692025-02-28T12:04:40.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45702025-02-28T12:04:40.710ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45712025-02-28T12:04:40.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45722025-02-28T12:04:40.714ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
45732025-02-28T12:04:40.714ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
45742025-02-28T12:04:40.714ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "3a87f4a3-2a58-4ef4-a79b-58a0cbc132d2", "date": "Fri, 28 Feb 2025 12:04:40 GMT"} })
45752025-02-28T12:04:40.717ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 257.753502ms iteration = 12 status = {}
45762025-02-28T12:04:40.780ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
45772025-02-28T12:04:40.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
45782025-02-28T12:04:40.782ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
45792025-02-28T12:04:40.783ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45802025-02-28T12:04:40.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
45812025-02-28T12:04:40.786ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
45822025-02-28T12:04:40.786ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
45832025-02-28T12:04:40.786ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
45842025-02-28T12:04:40.786ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
45852025-02-28T12:04:40.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
45862025-02-28T12:04:40.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
45872025-02-28T12:04:40.788ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45882025-02-28T12:04:40.791ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
45892025-02-28T12:04:40.791ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
45902025-02-28T12:04:40.791ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
45912025-02-28T12:04:40.798ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
45922025-02-28T12:04:40.798ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "03675aa0-25ba-4d64-abdd-3164887cc68d", "content-length": "134", "date": "Fri, 28 Feb 2025 12:04:40 GMT"} })
45932025-02-28T12:04:40.798ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
45942025-02-28T12:04:40.798ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
45952025-02-28T12:04:40.799ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
45962025-02-28T12:04:40.799ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
45972025-02-28T12:04:40.799ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
45982025-02-28T12:04:40.799ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
45992025-02-28T12:04:40.799ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
46002025-02-28T12:04:40.799ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
46012025-02-28T12:04:40.799ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
46022025-02-28T12:04:40.799ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
46032025-02-28T12:04:40.799ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
46042025-02-28T12:04:40.799ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46052025-02-28T12:04:40.800ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46062025-02-28T12:04:40.802ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
46072025-02-28T12:04:40.802ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "775e7ebb-af75-4b6e-85d3-955be9a2c2c5", "content-length": "2", "date": "Fri, 28 Feb 2025 12:04:40 GMT"} })
46082025-02-28T12:04:40.802ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
46092025-02-28T12:04:40.802ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bcd66b1-637b-421f-a481-4391c94b8e56", "content-length": "51", "date": "Fri, 28 Feb 2025 12:04:40 GMT"} })
46102025-02-28T12:04:40.802ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
46112025-02-28T12:04:40.802ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
46122025-02-28T12:04:40.843ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
46132025-02-28T12:04:40.950ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "c0809b59-00d8-4f0d-8512-00b1a25fcd63", "date": "Fri, 28 Feb 2025 12:04:40 GMT"} })
46142025-02-28T12:04:40.950ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
46152025-02-28T12:04:40.950ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
46162025-02-28T12:04:40.951ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c843a0dc-cf3a-4717-a034-a90343a6af0c", "content-length": "682", "date": "Fri, 28 Feb 2025 12:04:40 GMT"} })
46172025-02-28T12:04:40.951ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46182025-02-28T12:04:40.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46192025-02-28T12:04:40.951ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "22ea9dc9-a5e2-4fa3-928b-b3fbdf3b277d", "content-length": "691", "date": "Fri, 28 Feb 2025 12:04:40 GMT"} })
46202025-02-28T12:04:40.951ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
46212025-02-28T12:04:40.951ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
46222025-02-28T12:04:40.951ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
46232025-02-28T12:04:40.953ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 493.625367ms iteration = 12 status = {}
46242025-02-28T12:04:40.955ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46252025-02-28T12:04:40.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46262025-02-28T12:04:40.959ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46272025-02-28T12:04:40.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46282025-02-28T12:04:40.961ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46292025-02-28T12:04:40.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46302025-02-28T12:04:40.976ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 516.403016ms iteration = 12 status = {}
46312025-02-28T12:04:44.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 417 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 5156f497-82fa-40f5-9005-75adaa5be0de response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
46322025-02-28T12:04:50.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 35 reason = Timeout
46332025-02-28T12:04:50.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46342025-02-28T12:04:50.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46352025-02-28T12:04:50.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46362025-02-28T12:04:50.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 35 reason = Timeout
46372025-02-28T12:04:50.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
46382025-02-28T12:04:50.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
46392025-02-28T12:04:50.461ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
46402025-02-28T12:04:50.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
46412025-02-28T12:04:50.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
46422025-02-28T12:04:50.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46432025-02-28T12:04:50.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46442025-02-28T12:04:50.463ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.787282ms iteration = 35 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
46452025-02-28T12:04:50.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46462025-02-28T12:04:50.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46472025-02-28T12:04:50.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46482025-02-28T12:04:50.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
46492025-02-28T12:04:50.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46502025-02-28T12:04:50.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46512025-02-28T12:04:50.478ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
46522025-02-28T12:04:50.478ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.979837ms iteration = 35 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
46532025-02-28T12:04:54.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 278 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = f1d942b1-2f4c-4cd7-b48e-c86e63009050 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
46542025-02-28T12:05:00.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 36 reason = Timeout
46552025-02-28T12:05:00.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46562025-02-28T12:05:00.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46572025-02-28T12:05:00.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46582025-02-28T12:05:00.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 36 reason = Timeout
46592025-02-28T12:05:00.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
46602025-02-28T12:05:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
46612025-02-28T12:05:00.461ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
46622025-02-28T12:05:00.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
46632025-02-28T12:05:00.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
46642025-02-28T12:05:00.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46652025-02-28T12:05:00.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46662025-02-28T12:05:00.463ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.494735ms iteration = 36 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
46672025-02-28T12:05:00.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46682025-02-28T12:05:00.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46692025-02-28T12:05:00.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46702025-02-28T12:05:00.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
46712025-02-28T12:05:00.468ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46722025-02-28T12:05:00.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46732025-02-28T12:05:00.482ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
46742025-02-28T12:05:00.482ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.050755ms iteration = 36 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
46752025-02-28T12:05:04.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 274 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = fbce08d3-3391-41fa-bac9-4bb983ba9d33 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
46762025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 7 reason = Timeout
46772025-02-28T12:05:10.455ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-28 11:55:10.455245500 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
46782025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
46792025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 9 reason = Timeout
46802025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 13 reason = Timeout
46812025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 7 reason = Timeout
46822025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
46832025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
46842025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 13 reason = Timeout
46852025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 9 reason = Timeout
46862025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
46872025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 7 reason = Timeout
46882025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
46892025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
46902025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 7 reason = Timeout
46912025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
46922025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
46932025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 7 reason = Timeout
46942025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 7 reason = Timeout
46952025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
46962025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
46972025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
46982025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
46992025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
47002025-02-28T12:05:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
47012025-02-28T12:05:10.455ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5cefabf1-6831-46ff-98eb-1a3b5abad8b9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1145, tv_nsec: 91370658 } })
47022025-02-28T12:05:10.456ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5ebff2d9-48b6-4fe6-b852-684bb14b4955.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bddfb0f9-1d7b-45e4-b3a5-35aa6792956b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1145, tv_nsec: 91546292 } })
47032025-02-28T12:05:10.456ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
47042025-02-28T12:05:10.456ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.050516ms iteration = 7 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
47052025-02-28T12:05:10.456ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
47062025-02-28T12:05:10.456ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.198919ms iteration = 7 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
47072025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
47082025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 37 reason = Timeout
47092025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 8 reason = Timeout
47102025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47112025-02-28T12:05:10.457ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
47122025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 9.78\u{b5}s iteration = 8 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
47132025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 13 reason = Timeout
47142025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 4 reason = Timeout
47152025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
47162025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
47172025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47182025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
47192025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
47202025-02-28T12:05:10.457ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-28 11:55:10.455245500 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
47212025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
47222025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 2.698045ms iteration = 7 status = {"expiration":"2025-02-28T11:55:10.455245500Z","pruned":[]}
47232025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
47242025-02-28T12:05:10.457ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
47252025-02-28T12:05:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
47262025-02-28T12:05:10.458ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
47272025-02-28T12:05:10.458ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47282025-02-28T12:05:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
47292025-02-28T12:05:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
47302025-02-28T12:05:10.458ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
47312025-02-28T12:05:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
47322025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 7 reason = Timeout
47332025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 13 reason = Timeout
47342025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
47352025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
47362025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 13 reason = Timeout
47372025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 13 reason = Timeout
47382025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
47392025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47402025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 13 reason = Timeout
47412025-02-28T12:05:10.459ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
47422025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 7 reason = Timeout
47432025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
47442025-02-28T12:05:10.459ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
47452025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 13 reason = Timeout
47462025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
47472025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
47482025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 37 reason = Timeout
47492025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 13 reason = Timeout
47502025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
47512025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
47522025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 13 reason = Timeout
47532025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 14 reason = Timeout
47542025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
47552025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
47562025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 13 reason = Timeout
47572025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
47582025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
47592025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
47602025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
47612025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
47622025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47632025-02-28T12:05:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
47642025-02-28T12:05:10.459ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1145, tv_nsec: 95368845 } })
47652025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
47662025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
47672025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
47682025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
47692025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
47702025-02-28T12:05:10.460ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1145, tv_nsec: 95505508 } })
47712025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
47722025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
47732025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
47742025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
47752025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
47762025-02-28T12:05:10.460ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
47772025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 13 reason = Timeout
47782025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
47792025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 13 reason = Timeout
47802025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
47812025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 7 reason = Timeout
47822025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 7 reason = Timeout
47832025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
47842025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
47852025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 13 reason = Timeout
47862025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
47872025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 13 reason = Timeout
47882025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
47892025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 13 reason = Timeout
47902025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
47912025-02-28T12:05:10.460ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
47922025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
47932025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
47942025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
47952025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
47962025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
47972025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
47982025-02-28T12:05:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
47992025-02-28T12:05:10.462ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48002025-02-28T12:05:10.462ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48012025-02-28T12:05:10.463ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.847411ms iteration = 13 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
48022025-02-28T12:05:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
48032025-02-28T12:05:10.465ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
48042025-02-28T12:05:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
48052025-02-28T12:05:10.465ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48062025-02-28T12:05:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
48072025-02-28T12:05:10.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
48082025-02-28T12:05:10.466ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
48092025-02-28T12:05:10.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
48102025-02-28T12:05:10.467ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48112025-02-28T12:05:10.467ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48122025-02-28T12:05:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
48132025-02-28T12:05:10.469ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
48142025-02-28T12:05:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
48152025-02-28T12:05:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
48162025-02-28T12:05:10.470ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48172025-02-28T12:05:10.471ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48182025-02-28T12:05:10.471ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48192025-02-28T12:05:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
48202025-02-28T12:05:10.472ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
48212025-02-28T12:05:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
48222025-02-28T12:05:10.472ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48232025-02-28T12:05:10.474ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
48242025-02-28T12:05:10.474ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48252025-02-28T12:05:10.474ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
48262025-02-28T12:05:10.474ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
48272025-02-28T12:05:10.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
48282025-02-28T12:05:10.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48292025-02-28T12:05:10.477ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
48302025-02-28T12:05:10.477ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
48312025-02-28T12:05:10.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
48322025-02-28T12:05:10.478ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48332025-02-28T12:05:10.480ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48342025-02-28T12:05:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
48352025-02-28T12:05:10.480ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
48362025-02-28T12:05:10.480ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48372025-02-28T12:05:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
48382025-02-28T12:05:10.482ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
48392025-02-28T12:05:10.482ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48402025-02-28T12:05:10.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48412025-02-28T12:05:10.483ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
48422025-02-28T12:05:10.483ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
48432025-02-28T12:05:10.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
48442025-02-28T12:05:10.489ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
48452025-02-28T12:05:10.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
48462025-02-28T12:05:10.494ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 34.178546ms iteration = 7 status = {"errors":[],"found_port_ok":[]}
48472025-02-28T12:05:10.501ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
48482025-02-28T12:05:10.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
48492025-02-28T12:05:10.502ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 44.895794ms iteration = 4 status = {"nsuccess":5}
48502025-02-28T12:05:10.504ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1145, tv_nsec: 139378427 } })
48512025-02-28T12:05:10.504ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
48522025-02-28T12:05:10.504ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48532025-02-28T12:05:10.506ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
48542025-02-28T12:05:10.506ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48552025-02-28T12:05:10.506ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48562025-02-28T12:05:10.514ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48572025-02-28T12:05:10.515ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48582025-02-28T12:05:10.516ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
48592025-02-28T12:05:10.516ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
48602025-02-28T12:05:10.516ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
48612025-02-28T12:05:10.516ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
48622025-02-28T12:05:10.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
48632025-02-28T12:05:10.517ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
48642025-02-28T12:05:10.517ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
48652025-02-28T12:05:10.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
48662025-02-28T12:05:10.518ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
48672025-02-28T12:05:10.518ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48682025-02-28T12:05:10.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48692025-02-28T12:05:10.520ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48702025-02-28T12:05:10.524ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
48712025-02-28T12:05:10.524ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
48722025-02-28T12:05:10.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
48732025-02-28T12:05:10.525ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
48742025-02-28T12:05:10.525ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
48752025-02-28T12:05:10.525ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
48762025-02-28T12:05:10.525ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
48772025-02-28T12:05:10.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
48782025-02-28T12:05:10.528ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48792025-02-28T12:05:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48802025-02-28T12:05:10.532ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
48812025-02-28T12:05:10.532ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48822025-02-28T12:05:10.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
48832025-02-28T12:05:10.535ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48842025-02-28T12:05:10.535ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
48852025-02-28T12:05:10.535ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
48862025-02-28T12:05:10.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
48872025-02-28T12:05:10.535ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
48882025-02-28T12:05:10.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
48892025-02-28T12:05:10.536ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
48902025-02-28T12:05:10.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
48912025-02-28T12:05:10.537ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
48922025-02-28T12:05:10.537ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
48932025-02-28T12:05:10.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
48942025-02-28T12:05:10.542ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
48952025-02-28T12:05:10.542ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
48962025-02-28T12:05:10.542ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 82.362488ms iteration = 7 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
48972025-02-28T12:05:10.543ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
48982025-02-28T12:05:10.543ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 82.918202ms iteration = 13 status = {"errors":[],"requests_created_ok":[]}
48992025-02-28T12:05:10.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
49002025-02-28T12:05:10.548ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
49012025-02-28T12:05:10.548ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
49022025-02-28T12:05:10.548ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
49032025-02-28T12:05:10.548ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
49042025-02-28T12:05:10.549ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 89.431139ms iteration = 13 status = {"errors":[],"garbage_collect_requested":[]}
49052025-02-28T12:05:10.550ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 89.850049ms iteration = 13 status = {"errors":[],"finish_invoked_ok":[]}
49062025-02-28T12:05:10.552ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
49072025-02-28T12:05:10.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
49082025-02-28T12:05:10.554ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
49092025-02-28T12:05:10.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
49102025-02-28T12:05:10.556ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
49112025-02-28T12:05:10.556ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 96.454609ms iteration = 13 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
49122025-02-28T12:05:10.558ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
49132025-02-28T12:05:10.558ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
49142025-02-28T12:05:10.565ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
49152025-02-28T12:05:10.566ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
49162025-02-28T12:05:10.604ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
49172025-02-28T12:05:10.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
49182025-02-28T12:05:10.620ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
49192025-02-28T12:05:10.620ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
49202025-02-28T12:05:10.621ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bca21c52-e67d-4f7c-b08a-061a31f3ac9f", "content-length": "26", "date": "Fri, 28 Feb 2025 12:05:10 GMT"} })
49212025-02-28T12:05:10.621ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
49222025-02-28T12:05:10.624ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
49232025-02-28T12:05:10.630ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
49242025-02-28T12:05:10.693ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
49252025-02-28T12:05:10.693ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
49262025-02-28T12:05:10.694ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "924baf5b-dd0f-41dc-bc9e-44d4c4646397", "content-length": "26", "date": "Fri, 28 Feb 2025 12:05:10 GMT"} })
49272025-02-28T12:05:10.694ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
49282025-02-28T12:05:10.696ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
49292025-02-28T12:05:10.748ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
49302025-02-28T12:05:10.748ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
49312025-02-28T12:05:10.748ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "e02d28cb-6ce1-4815-88e5-7c7c0d50b13d", "content-length": "372", "date": "Fri, 28 Feb 2025 12:05:10 GMT"} })
49322025-02-28T12:05:10.748ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
49332025-02-28T12:05:10.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
49342025-02-28T12:05:10.754ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
49352025-02-28T12:05:10.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
49362025-02-28T12:05:10.757ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49372025-02-28T12:05:10.762ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
49382025-02-28T12:05:10.762ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
49392025-02-28T12:05:10.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
49402025-02-28T12:05:10.765ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 305.71657ms iteration = 37 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
49412025-02-28T12:05:10.768ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
49422025-02-28T12:05:10.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
49432025-02-28T12:05:10.771ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
49442025-02-28T12:05:10.772ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
49452025-02-28T12:05:10.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
49462025-02-28T12:05:10.780ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49472025-02-28T12:05:10.791ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
49482025-02-28T12:05:10.791ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
49492025-02-28T12:05:10.791ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
49502025-02-28T12:05:10.803ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 344.106557ms iteration = 13 status = 0
49512025-02-28T12:05:10.829ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 368.760422ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
49522025-02-28T12:05:10.832ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
49532025-02-28T12:05:10.832ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
49542025-02-28T12:05:10.832ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
49552025-02-28T12:05:10.832ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49562025-02-28T12:05:10.832ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
49572025-02-28T12:05:10.834ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
49582025-02-28T12:05:10.834ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
49592025-02-28T12:05:10.834ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 374.89987ms iteration = 7 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
49602025-02-28T12:05:10.834ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49612025-02-28T12:05:10.836ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
49622025-02-28T12:05:10.836ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49632025-02-28T12:05:10.837ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49642025-02-28T12:05:10.839ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
49652025-02-28T12:05:10.839ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
49662025-02-28T12:05:10.839ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
49672025-02-28T12:05:10.840ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
49682025-02-28T12:05:10.840ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
49692025-02-28T12:05:10.842ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e484d0e-78c1-4221-874b-c1158c0049ae", "content-length": "26", "date": "Fri, 28 Feb 2025 12:05:10 GMT"} })
49702025-02-28T12:05:10.842ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
49712025-02-28T12:05:10.845ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
49722025-02-28T12:05:10.845ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
49732025-02-28T12:05:10.845ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
49742025-02-28T12:05:10.845ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
49752025-02-28T12:05:10.846ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 390.697231ms iteration = 13 status = {}
49762025-02-28T12:05:10.846ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
49772025-02-28T12:05:10.846ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
49782025-02-28T12:05:10.846ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-02-28 11:59:51.915187 UTC dns_group = internal generation = 1
49792025-02-28T12:05:10.846ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 390.930077ms iteration = 7 status = {"generation":1}
49802025-02-28T12:05:10.846ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49812025-02-28T12:05:10.847ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
49822025-02-28T12:05:10.849ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
49832025-02-28T12:05:10.849ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 0 version = 2
49842025-02-28T12:05:10.849ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
49852025-02-28T12:05:10.850ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
49862025-02-28T12:05:10.850ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
49872025-02-28T12:05:10.850ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
49882025-02-28T12:05:10.850ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
49892025-02-28T12:05:10.850ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
49902025-02-28T12:05:10.850ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 390.696872ms iteration = 13 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
49912025-02-28T12:05:10.850ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
49922025-02-28T12:05:10.852ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49932025-02-28T12:05:10.852ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 392.250469ms iteration = 13 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
49942025-02-28T12:05:10.853ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 394.000491ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
49952025-02-28T12:05:10.853ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49962025-02-28T12:05:10.856ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
49972025-02-28T12:05:10.856ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
49982025-02-28T12:05:10.857ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
49992025-02-28T12:05:10.857ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
50002025-02-28T12:05:10.857ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
50012025-02-28T12:05:10.857ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
50022025-02-28T12:05:10.857ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
50032025-02-28T12:05:10.858ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "3ce967b1-d9f9-44d2-9444-f4cd26a5860a", "date": "Fri, 28 Feb 2025 12:05:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
50042025-02-28T12:05:10.858ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
50052025-02-28T12:05:10.859ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 1 version = 2
50062025-02-28T12:05:10.859ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
50072025-02-28T12:05:10.859ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-02-28 11:59:52.542241 UTC dns_group = external generation = 2
50082025-02-28T12:05:10.859ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 404.367111ms iteration = 7 status = {"generation":2}
50092025-02-28T12:05:10.860ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "cf6dbefe-63e8-4eeb-831c-fc93cb2ca9ba", "date": "Fri, 28 Feb 2025 12:05:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
50102025-02-28T12:05:10.860ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
50112025-02-28T12:05:10.867ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
50122025-02-28T12:05:10.867ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 412.1484ms iteration = 9 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
50132025-02-28T12:05:10.870ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50142025-02-28T12:05:10.872ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
50152025-02-28T12:05:10.872ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 415.350287ms iteration = 37 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
50162025-02-28T12:05:10.873ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
50172025-02-28T12:05:10.873ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50182025-02-28T12:05:10.873ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50192025-02-28T12:05:10.889ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
50202025-02-28T12:05:10.889ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
50212025-02-28T12:05:10.890ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "b9891e14-dbf1-41b7-910e-9f4ba4ff9a21", "date": "Fri, 28 Feb 2025 12:05:10 GMT"} })
50222025-02-28T12:05:10.890ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50232025-02-28T12:05:10.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50242025-02-28T12:05:10.894ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50252025-02-28T12:05:10.894ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50262025-02-28T12:05:10.897ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50272025-02-28T12:05:10.897ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50282025-02-28T12:05:10.902ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
50292025-02-28T12:05:10.902ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
50302025-02-28T12:05:10.902ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "99739207-dc4e-4359-a959-9a70b8023144", "date": "Fri, 28 Feb 2025 12:05:10 GMT"} })
50312025-02-28T12:05:10.904ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 444.638824ms iteration = 13 status = {}
50322025-02-28T12:05:10.910ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
50332025-02-28T12:05:10.910ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 450.935575ms iteration = 13 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
50342025-02-28T12:05:10.911ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-28T11:59:08.719399Z, time_modified: 2025-02-28T11:59:08.719399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }) }
50352025-02-28T12:05:10.911ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 456.130581ms iteration = 7 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]}},"default_endpoint":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
50362025-02-28T12:05:10.943ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
50372025-02-28T12:05:10.943ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 483.571794ms iteration = 7 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
50382025-02-28T12:05:10.960ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
50392025-02-28T12:05:10.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
50402025-02-28T12:05:10.963ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50412025-02-28T12:05:10.965ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
50422025-02-28T12:05:10.965ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
50432025-02-28T12:05:10.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
50442025-02-28T12:05:10.966ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50452025-02-28T12:05:10.968ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
50462025-02-28T12:05:10.968ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
50472025-02-28T12:05:10.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
50482025-02-28T12:05:10.995ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
50492025-02-28T12:05:11.008ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 548.560513ms iteration = 14 status = {"physical_disks_added":0}
50502025-02-28T12:05:11.014ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
50512025-02-28T12:05:11.014ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "303833f9-165f-4241-a90a-2bdc2aaf89a3", "content-length": "1", "date": "Fri, 28 Feb 2025 12:05:10 GMT"} })
50522025-02-28T12:05:11.016ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
50532025-02-28T12:05:11.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
50542025-02-28T12:05:11.019ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 563.922993ms iteration = 13 status = null
50552025-02-28T12:05:11.060ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
50562025-02-28T12:05:11.061ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "047e6f0f-a043-40ed-ae5d-5406c43a22c8", "date": "Fri, 28 Feb 2025 12:05:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
50572025-02-28T12:05:11.061ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
50582025-02-28T12:05:11.064ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "9822d8ee-dbd3-4d51-a165-6de1cc1cfe6f", "date": "Fri, 28 Feb 2025 12:05:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
50592025-02-28T12:05:11.064ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
50602025-02-28T12:05:11.066ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "1731f608-fead-4d04-989b-fab82bed4ae0", "date": "Fri, 28 Feb 2025 12:05:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
50612025-02-28T12:05:11.066ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
50622025-02-28T12:05:11.069ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
50632025-02-28T12:05:11.069ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 614.029263ms iteration = 9 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
50642025-02-28T12:05:11.109ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50652025-02-28T12:05:11.109ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50662025-02-28T12:05:11.112ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
50672025-02-28T12:05:11.112ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50682025-02-28T12:05:11.112ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50692025-02-28T12:05:11.125ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
50702025-02-28T12:05:11.125ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "35997565-63a1-41ff-a579-5018a8637ac8", "content-length": "134", "date": "Fri, 28 Feb 2025 12:05:11 GMT"} })
50712025-02-28T12:05:11.125ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
50722025-02-28T12:05:11.125ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
50732025-02-28T12:05:11.126ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
50742025-02-28T12:05:11.126ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
50752025-02-28T12:05:11.126ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
50762025-02-28T12:05:11.126ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
50772025-02-28T12:05:11.126ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
50782025-02-28T12:05:11.126ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
50792025-02-28T12:05:11.126ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
50802025-02-28T12:05:11.126ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
50812025-02-28T12:05:11.126ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
50822025-02-28T12:05:11.126ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50832025-02-28T12:05:11.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50842025-02-28T12:05:11.129ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
50852025-02-28T12:05:11.129ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "db2e2c5b-1256-4024-a0b1-9dbf2390b2a5", "content-length": "2", "date": "Fri, 28 Feb 2025 12:05:11 GMT"} })
50862025-02-28T12:05:11.129ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
50872025-02-28T12:05:11.129ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "0abbdd65-047f-4cd9-bd18-4cc557e55043", "content-length": "51", "date": "Fri, 28 Feb 2025 12:05:11 GMT"} })
50882025-02-28T12:05:11.130ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
50892025-02-28T12:05:11.130ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
50902025-02-28T12:05:11.135ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
50912025-02-28T12:05:11.167ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
50922025-02-28T12:05:11.167ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::a]:12221
50932025-02-28T12:05:11.210ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
50942025-02-28T12:05:11.210ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::a]:47912
50952025-02-28T12:05:11.210ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:47912 req_id = 763e8975-3a82-45c6-893c-c45ed25de6f4 roles = RoleSet { roles: {} } uri = /metrics/producers
50962025-02-28T12:05:11.211ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:47912 req_id = 763e8975-3a82-45c6-893c-c45ed25de6f4 resource = Database result = Ok(()) uri = /metrics/producers
50972025-02-28T12:05:11.253ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "8ca2acc1-b58f-44e4-a6a9-7ff5d106bbd8", "date": "Fri, 28 Feb 2025 12:05:11 GMT"} })
50982025-02-28T12:05:11.253ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
50992025-02-28T12:05:11.253ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
51002025-02-28T12:05:11.254ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "1577321c-a869-4015-b990-fb0b88297707", "content-length": "682", "date": "Fri, 28 Feb 2025 12:05:11 GMT"} })
51012025-02-28T12:05:11.254ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51022025-02-28T12:05:11.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51032025-02-28T12:05:11.255ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "7baceaf1-ba11-4fd0-9c97-5c8e23467c57", "content-length": "691", "date": "Fri, 28 Feb 2025 12:05:10 GMT"} })
51042025-02-28T12:05:11.255ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
51052025-02-28T12:05:11.255ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
51062025-02-28T12:05:11.255ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
51072025-02-28T12:05:11.256ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 797.183645ms iteration = 13 status = {}
51082025-02-28T12:05:11.259ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51092025-02-28T12:05:11.259ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51102025-02-28T12:05:11.263ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51112025-02-28T12:05:11.263ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51122025-02-28T12:05:11.266ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51132025-02-28T12:05:11.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51142025-02-28T12:05:11.270ZINFOnexus: assigned collector to new producer collector_id = a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79 file = nexus/src/app/oximeter.rs:76 producer_id = c1b34bc5-1c2f-4d3c-808c-b7a441aac295
51152025-02-28T12:05:11.270ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 59317 local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:47912 req_id = 763e8975-3a82-45c6-893c-c45ed25de6f4 response_code = 201 uri = /metrics/producers
51162025-02-28T12:05:11.270ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "763e8975-3a82-45c6-893c-c45ed25de6f4", "content-length": "41", "date": "Fri, 28 Feb 2025 12:05:11 GMT"} })
51172025-02-28T12:05:11.271ZDEBGnexus (producer-registration-task): registered with nexus successfully
51182025-02-28T12:05:11.271ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
51192025-02-28T12:05:11.278ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 818.572101ms iteration = 13 status = {}
51202025-02-28T12:05:14.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 448 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 320d8d34-ee25-4439-a868-b82c816131cc response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
51212025-02-28T12:05:20.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 38 reason = Timeout
51222025-02-28T12:05:20.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51232025-02-28T12:05:20.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51242025-02-28T12:05:20.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 38 reason = Timeout
51252025-02-28T12:05:20.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
51262025-02-28T12:05:20.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51272025-02-28T12:05:20.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
51282025-02-28T12:05:20.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
51292025-02-28T12:05:20.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51302025-02-28T12:05:20.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51312025-02-28T12:05:20.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51322025-02-28T12:05:20.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51332025-02-28T12:05:20.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51342025-02-28T12:05:20.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
51352025-02-28T12:05:20.469ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51362025-02-28T12:05:20.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51372025-02-28T12:05:20.483ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
51382025-02-28T12:05:20.483ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.67847ms iteration = 38 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
51392025-02-28T12:05:20.497ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
51402025-02-28T12:05:20.498ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
51412025-02-28T12:05:20.537ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 77.779088ms iteration = 38 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
51422025-02-28T12:05:24.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 266 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = f452d997-58e7-460f-829c-45eb50fe0faf response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
51432025-02-28T12:05:30.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 39 reason = Timeout
51442025-02-28T12:05:30.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51452025-02-28T12:05:30.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51462025-02-28T12:05:30.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 39 reason = Timeout
51472025-02-28T12:05:30.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
51482025-02-28T12:05:30.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51492025-02-28T12:05:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
51502025-02-28T12:05:30.461ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
51512025-02-28T12:05:30.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
51522025-02-28T12:05:30.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51532025-02-28T12:05:30.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
51542025-02-28T12:05:30.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51552025-02-28T12:05:30.464ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.390546ms iteration = 39 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
51562025-02-28T12:05:30.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51572025-02-28T12:05:30.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51582025-02-28T12:05:30.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51592025-02-28T12:05:30.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
51602025-02-28T12:05:30.469ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51612025-02-28T12:05:30.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51622025-02-28T12:05:30.481ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
51632025-02-28T12:05:30.481ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.495077ms iteration = 39 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
51642025-02-28T12:05:34.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 298 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = f0067486-254e-4fc0-9d19-33ae2c0307ff response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
51652025-02-28T12:05:40.454ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
51662025-02-28T12:05:40.454ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
51672025-02-28T12:05:40.454ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
51682025-02-28T12:05:40.454ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
51692025-02-28T12:05:40.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
51702025-02-28T12:05:40.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
51712025-02-28T12:05:40.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
51722025-02-28T12:05:40.457ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
51732025-02-28T12:05:40.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51742025-02-28T12:05:40.457ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1175, tv_nsec: 92901305 } })
51752025-02-28T12:05:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51762025-02-28T12:05:40.457ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
51772025-02-28T12:05:40.457ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1175, tv_nsec: 93236323 } })
51782025-02-28T12:05:40.457ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
51792025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 40 reason = Timeout
51802025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
51812025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
51822025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
51832025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 14 reason = Timeout
51842025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
51852025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 14 reason = Timeout
51862025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51872025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
51882025-02-28T12:05:40.459ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
51892025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51902025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 14 reason = Timeout
51912025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
51922025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 15 reason = Timeout
51932025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
51942025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
51952025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
51962025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 14 reason = Timeout
51972025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
51982025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 14 reason = Timeout
51992025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52002025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 14 reason = Timeout
52012025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
52022025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52032025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
52042025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52052025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
52062025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
52072025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
52082025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
52092025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
52102025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52112025-02-28T12:05:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
52122025-02-28T12:05:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 14 reason = Timeout
52132025-02-28T12:05:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 14 reason = Timeout
52142025-02-28T12:05:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
52152025-02-28T12:05:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
52162025-02-28T12:05:40.460ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 14 reason = Timeout
52172025-02-28T12:05:40.460ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
52182025-02-28T12:05:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 14 reason = Timeout
52192025-02-28T12:05:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
52202025-02-28T12:05:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 14 reason = Timeout
52212025-02-28T12:05:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
52222025-02-28T12:05:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
52232025-02-28T12:05:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
52242025-02-28T12:05:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
52252025-02-28T12:05:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
52262025-02-28T12:05:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
52272025-02-28T12:05:40.461ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.939855ms iteration = 14 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
52282025-02-28T12:05:40.461ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52292025-02-28T12:05:40.461ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52302025-02-28T12:05:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
52312025-02-28T12:05:40.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52322025-02-28T12:05:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52332025-02-28T12:05:40.463ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
52342025-02-28T12:05:40.463ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52352025-02-28T12:05:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52362025-02-28T12:05:40.464ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52372025-02-28T12:05:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
52382025-02-28T12:05:40.464ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52392025-02-28T12:05:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52402025-02-28T12:05:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
52412025-02-28T12:05:40.465ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
52422025-02-28T12:05:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
52432025-02-28T12:05:40.466ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52442025-02-28T12:05:40.466ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52452025-02-28T12:05:40.467ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52462025-02-28T12:05:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
52472025-02-28T12:05:40.468ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
52482025-02-28T12:05:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
52492025-02-28T12:05:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
52502025-02-28T12:05:40.469ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52512025-02-28T12:05:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52522025-02-28T12:05:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
52532025-02-28T12:05:40.469ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
52542025-02-28T12:05:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
52552025-02-28T12:05:40.476ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52562025-02-28T12:05:40.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52572025-02-28T12:05:40.477ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52582025-02-28T12:05:40.478ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 17.617105ms iteration = 14 status = {"errors":[],"requests_created_ok":[]}
52592025-02-28T12:05:40.478ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
52602025-02-28T12:05:40.478ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52612025-02-28T12:05:40.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52622025-02-28T12:05:40.479ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52632025-02-28T12:05:40.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52642025-02-28T12:05:40.480ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
52652025-02-28T12:05:40.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
52662025-02-28T12:05:40.481ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1175, tv_nsec: 117249253 } })
52672025-02-28T12:05:40.482ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
52682025-02-28T12:05:40.486ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
52692025-02-28T12:05:40.486ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 27.424192ms iteration = 14 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
52702025-02-28T12:05:40.496ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
52712025-02-28T12:05:40.496ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
52722025-02-28T12:05:40.496ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
52732025-02-28T12:05:40.496ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
52742025-02-28T12:05:40.501ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52752025-02-28T12:05:40.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
52762025-02-28T12:05:40.505ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 44.652508ms iteration = 14 status = {"errors":[],"garbage_collect_requested":[]}
52772025-02-28T12:05:40.505ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
52782025-02-28T12:05:40.506ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
52792025-02-28T12:05:40.513ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
52802025-02-28T12:05:40.513ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
52812025-02-28T12:05:40.515ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 56.153175ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
52822025-02-28T12:05:40.516ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 56.030742ms iteration = 14 status = {"errors":[],"finish_invoked_ok":[]}
52832025-02-28T12:05:40.519ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
52842025-02-28T12:05:40.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
52852025-02-28T12:05:40.521ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52862025-02-28T12:05:40.521ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
52872025-02-28T12:05:40.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52882025-02-28T12:05:40.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
52892025-02-28T12:05:40.523ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52902025-02-28T12:05:40.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52912025-02-28T12:05:40.527ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
52922025-02-28T12:05:40.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
52932025-02-28T12:05:40.527ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52942025-02-28T12:05:40.527ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52952025-02-28T12:05:40.529ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
52962025-02-28T12:05:40.529ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
52972025-02-28T12:05:40.529ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
52982025-02-28T12:05:40.529ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52992025-02-28T12:05:40.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
53002025-02-28T12:05:40.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
53012025-02-28T12:05:40.531ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
53022025-02-28T12:05:40.531ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 72.48182ms iteration = 14 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
53032025-02-28T12:05:40.532ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 72.758456ms iteration = 14 status = 0
53042025-02-28T12:05:40.535ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53052025-02-28T12:05:40.537ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53062025-02-28T12:05:40.538ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
53072025-02-28T12:05:40.538ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
53082025-02-28T12:05:40.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
53092025-02-28T12:05:40.538ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
53102025-02-28T12:05:40.538ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
53112025-02-28T12:05:40.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
53122025-02-28T12:05:40.544ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
53132025-02-28T12:05:40.544ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
53142025-02-28T12:05:40.544ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
53152025-02-28T12:05:40.544ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
53162025-02-28T12:05:40.550ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 90.379732ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
53172025-02-28T12:05:40.551ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 92.031342ms iteration = 40 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
53182025-02-28T12:05:40.555ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
53192025-02-28T12:05:40.555ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 98.117099ms iteration = 40 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
53202025-02-28T12:05:40.572ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 112.311882ms iteration = 14 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
53212025-02-28T12:05:40.583ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
53222025-02-28T12:05:40.583ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
53232025-02-28T12:05:40.583ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 123.893421ms iteration = 14 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
53242025-02-28T12:05:40.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53252025-02-28T12:05:40.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
53262025-02-28T12:05:40.601ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
53272025-02-28T12:05:40.601ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
53282025-02-28T12:05:40.602ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "247b268f-b267-4c99-8daa-0cdb197a081c", "content-length": "26", "date": "Fri, 28 Feb 2025 12:05:40 GMT"} })
53292025-02-28T12:05:40.602ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
53302025-02-28T12:05:40.603ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
53312025-02-28T12:05:40.655ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 196.327359ms iteration = 15 status = {"physical_disks_added":0}
53322025-02-28T12:05:40.662ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53332025-02-28T12:05:40.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
53342025-02-28T12:05:40.666ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53352025-02-28T12:05:40.667ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
53362025-02-28T12:05:40.667ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
53372025-02-28T12:05:40.667ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ec7d22f-9bd4-4ff3-a67d-e731c46076a5", "content-length": "26", "date": "Fri, 28 Feb 2025 12:05:40 GMT"} })
53382025-02-28T12:05:40.667ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
53392025-02-28T12:05:40.668ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
53402025-02-28T12:05:40.668ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53412025-02-28T12:05:40.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
53422025-02-28T12:05:40.670ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
53432025-02-28T12:05:40.670ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53442025-02-28T12:05:40.673ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
53452025-02-28T12:05:40.673ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53462025-02-28T12:05:40.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
53472025-02-28T12:05:40.729ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
53482025-02-28T12:05:40.729ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
53492025-02-28T12:05:40.729ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c78c499-a970-4bf7-9b7b-2ebedcabc4f5", "content-length": "26", "date": "Fri, 28 Feb 2025 12:05:40 GMT"} })
53502025-02-28T12:05:40.729ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
53512025-02-28T12:05:40.732ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
53522025-02-28T12:05:40.780ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
53532025-02-28T12:05:40.780ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
53542025-02-28T12:05:40.780ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "c612cf9d-81c1-4d80-9f8e-c763198657b8", "content-length": "372", "date": "Fri, 28 Feb 2025 12:05:40 GMT"} })
53552025-02-28T12:05:40.780ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53562025-02-28T12:05:40.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53572025-02-28T12:05:40.782ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 328.51668ms iteration = 14 status = {}
53582025-02-28T12:05:40.801ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
53592025-02-28T12:05:40.801ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
53602025-02-28T12:05:40.801ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "1f7f61aa-57a4-4b92-9f97-c4c96e94bae1", "date": "Fri, 28 Feb 2025 12:05:40 GMT"} })
53612025-02-28T12:05:40.801ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53622025-02-28T12:05:40.802ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53632025-02-28T12:05:40.804ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53642025-02-28T12:05:40.804ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53652025-02-28T12:05:40.808ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53662025-02-28T12:05:40.808ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53672025-02-28T12:05:40.812ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
53682025-02-28T12:05:40.812ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
53692025-02-28T12:05:40.812ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "fdb00afe-6757-440b-8a47-b7a175a22c4d", "date": "Fri, 28 Feb 2025 12:05:40 GMT"} })
53702025-02-28T12:05:40.816ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 356.986097ms iteration = 14 status = {}
53712025-02-28T12:05:40.837ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
53722025-02-28T12:05:40.838ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "59ca3df0-f123-417c-a4cc-e70403b64eb4", "content-length": "1", "date": "Fri, 28 Feb 2025 12:05:40 GMT"} })
53732025-02-28T12:05:40.839ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
53742025-02-28T12:05:40.839ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
53752025-02-28T12:05:40.851ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
53762025-02-28T12:05:40.852ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "db613fb7-2578-4d8d-a524-6ac8948c7a72", "content-length": "691", "date": "Fri, 28 Feb 2025 12:05:40 GMT"} })
53772025-02-28T12:05:40.852ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
53782025-02-28T12:05:40.852ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
53792025-02-28T12:05:40.852ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
53802025-02-28T12:05:40.853ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 394.487913ms iteration = 14 status = {}
53812025-02-28T12:05:40.854ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 399.657268ms iteration = 14 status = null
53822025-02-28T12:05:40.873ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
53832025-02-28T12:05:40.873ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
53842025-02-28T12:05:40.875ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
53852025-02-28T12:05:40.875ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
53862025-02-28T12:05:40.875ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
53872025-02-28T12:05:40.885ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
53882025-02-28T12:05:40.885ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7862b7e-b2e1-4e3c-a80e-7135b3cf8597", "content-length": "134", "date": "Fri, 28 Feb 2025 12:05:40 GMT"} })
53892025-02-28T12:05:40.885ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
53902025-02-28T12:05:40.885ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
53912025-02-28T12:05:40.886ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
53922025-02-28T12:05:40.886ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
53932025-02-28T12:05:40.886ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
53942025-02-28T12:05:40.886ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
53952025-02-28T12:05:40.886ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
53962025-02-28T12:05:40.886ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
53972025-02-28T12:05:40.886ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
53982025-02-28T12:05:40.886ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
53992025-02-28T12:05:40.886ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
54002025-02-28T12:05:40.886ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54012025-02-28T12:05:40.886ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54022025-02-28T12:05:40.888ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
54032025-02-28T12:05:40.888ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "19de3c9d-2106-476c-8f21-646fe433ed88", "content-length": "2", "date": "Fri, 28 Feb 2025 12:05:40 GMT"} })
54042025-02-28T12:05:40.888ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
54052025-02-28T12:05:40.889ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f0b0fca-0430-425e-bf93-8ac7aa6d8be2", "content-length": "51", "date": "Fri, 28 Feb 2025 12:05:40 GMT"} })
54062025-02-28T12:05:40.889ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
54072025-02-28T12:05:40.889ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
54082025-02-28T12:05:40.989ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "4de39266-263a-40b7-822e-3752ad1a4477", "date": "Fri, 28 Feb 2025 12:05:40 GMT"} })
54092025-02-28T12:05:40.989ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
54102025-02-28T12:05:40.989ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
54112025-02-28T12:05:40.989ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "06889734-c510-4cfe-bce6-d266a0780d77", "content-length": "682", "date": "Fri, 28 Feb 2025 12:05:40 GMT"} })
54122025-02-28T12:05:40.989ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54132025-02-28T12:05:40.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54142025-02-28T12:05:40.993ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54152025-02-28T12:05:40.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54162025-02-28T12:05:40.997ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54172025-02-28T12:05:40.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54182025-02-28T12:05:40.999ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54192025-02-28T12:05:40.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54202025-02-28T12:05:41.007ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 547.980758ms iteration = 14 status = {}
54212025-02-28T12:05:44.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 443 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = b413119e-049c-49e4-9fbc-618dea5bb143 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
54222025-02-28T12:05:50.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
54232025-02-28T12:05:50.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54242025-02-28T12:05:50.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54252025-02-28T12:05:50.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54262025-02-28T12:05:50.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 41 reason = Timeout
54272025-02-28T12:05:50.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
54282025-02-28T12:05:50.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
54292025-02-28T12:05:50.461ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
54302025-02-28T12:05:50.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
54312025-02-28T12:05:50.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54322025-02-28T12:05:50.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
54332025-02-28T12:05:50.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54342025-02-28T12:05:50.463ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.189561ms iteration = 41 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
54352025-02-28T12:05:50.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54362025-02-28T12:05:50.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54372025-02-28T12:05:50.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54382025-02-28T12:05:50.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
54392025-02-28T12:05:50.468ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54402025-02-28T12:05:50.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54412025-02-28T12:05:50.483ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
54422025-02-28T12:05:50.483ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.627423ms iteration = 41 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
54432025-02-28T12:05:54.254ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::d]:62104
54442025-02-28T12:05:54.254ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 002a03a2-a50e-4730-82db-eab369f2b4ed roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
54452025-02-28T12:05:54.254ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 002a03a2-a50e-4730-82db-eab369f2b4ed resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
54462025-02-28T12:05:54.257ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3357 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 002a03a2-a50e-4730-82db-eab369f2b4ed response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
54472025-02-28T12:05:54.258ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 1346618c-8f53-4454-b441-35e7c42ccce5 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
54482025-02-28T12:05:54.258ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 1346618c-8f53-4454-b441-35e7c42ccce5 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
54492025-02-28T12:05:54.260ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2613 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 1346618c-8f53-4454-b441-35e7c42ccce5 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
54502025-02-28T12:05:54.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 247 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 6de0da6a-d2e0-4cfc-9ee7-13990854fb0c response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
54512025-02-28T12:06:00.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
54522025-02-28T12:06:00.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54532025-02-28T12:06:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54542025-02-28T12:06:00.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 42 reason = Timeout
54552025-02-28T12:06:00.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
54562025-02-28T12:06:00.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54572025-02-28T12:06:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
54582025-02-28T12:06:00.461ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
54592025-02-28T12:06:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
54602025-02-28T12:06:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
54612025-02-28T12:06:00.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54622025-02-28T12:06:00.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54632025-02-28T12:06:00.463ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.116069ms iteration = 42 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
54642025-02-28T12:06:00.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54652025-02-28T12:06:00.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54662025-02-28T12:06:00.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54672025-02-28T12:06:00.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
54682025-02-28T12:06:00.468ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54692025-02-28T12:06:00.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54702025-02-28T12:06:00.487ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
54712025-02-28T12:06:00.487ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 30.519807ms iteration = 42 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
54722025-02-28T12:06:04.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 441 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 606ab51f-f95a-44af-8b7e-af8bf6eb1203 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
54732025-02-28T12:06:09.254ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 02fa7a78-75f7-4783-8162-29eb95f5d5f3 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
54742025-02-28T12:06:09.255ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 02fa7a78-75f7-4783-8162-29eb95f5d5f3 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
54752025-02-28T12:06:09.257ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3019 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 02fa7a78-75f7-4783-8162-29eb95f5d5f3 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
54762025-02-28T12:06:09.258ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = e20b968b-0a7a-41e2-9f50-2810dbc08353 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
54772025-02-28T12:06:09.258ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = e20b968b-0a7a-41e2-9f50-2810dbc08353 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
54782025-02-28T12:06:09.260ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2585 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = e20b968b-0a7a-41e2-9f50-2810dbc08353 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
54792025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 8 reason = Timeout
54802025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
54812025-02-28T12:06:10.454ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-28 11:56:10.454035488 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
54822025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
54832025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
54842025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
54852025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
54862025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
54872025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
54882025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
54892025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
54902025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
54912025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
54922025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
54932025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
54942025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
54952025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
54962025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
54972025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
54982025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
54992025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
55002025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
55012025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
55022025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
55032025-02-28T12:06:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
55042025-02-28T12:06:10.454ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5cefabf1-6831-46ff-98eb-1a3b5abad8b9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1205, tv_nsec: 90361883 } })
55052025-02-28T12:06:10.454ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5ebff2d9-48b6-4fe6-b852-684bb14b4955.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bddfb0f9-1d7b-45e4-b3a5-35aa6792956b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1205, tv_nsec: 90412845 } })
55062025-02-28T12:06:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
55072025-02-28T12:06:10.455ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
55082025-02-28T12:06:10.455ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 983.864\u{b5}s iteration = 8 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
55092025-02-28T12:06:10.455ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
55102025-02-28T12:06:10.455ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.105037ms iteration = 8 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
55112025-02-28T12:06:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
55122025-02-28T12:06:10.456ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
55132025-02-28T12:06:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
55142025-02-28T12:06:10.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
55152025-02-28T12:06:10.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
55162025-02-28T12:06:10.457ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
55172025-02-28T12:06:10.457ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
55182025-02-28T12:06:10.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55192025-02-28T12:06:10.457ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 20.37\u{b5}s iteration = 9 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
55202025-02-28T12:06:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55212025-02-28T12:06:10.457ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1205, tv_nsec: 93097989 } })
55222025-02-28T12:06:10.457ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-28 11:56:10.454035488 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
55232025-02-28T12:06:10.457ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.554486ms iteration = 8 status = {"expiration":"2025-02-28T11:56:10.454035488Z","pruned":[]}
55242025-02-28T12:06:10.457ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
55252025-02-28T12:06:10.458ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1205, tv_nsec: 93934720 } })
55262025-02-28T12:06:10.458ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
55272025-02-28T12:06:10.458ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55282025-02-28T12:06:10.458ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55292025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 15 reason = Timeout
55302025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
55312025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 8 reason = Timeout
55322025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
55332025-02-28T12:06:10.459ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
55342025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
55352025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
55362025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 15 reason = Timeout
55372025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
55382025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
55392025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 15 reason = Timeout
55402025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
55412025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 15 reason = Timeout
55422025-02-28T12:06:10.459ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
55432025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
55442025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
55452025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
55462025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 15 reason = Timeout
55472025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55482025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 16 reason = Timeout
55492025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
55502025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 15 reason = Timeout
55512025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
55522025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55532025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 43 reason = Timeout
55542025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55552025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 8 reason = Timeout
55562025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
55572025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
55582025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
55592025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
55602025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
55612025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
55622025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
55632025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
55642025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55652025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
55662025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55672025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
55682025-02-28T12:06:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
55692025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 15 reason = Timeout
55702025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 15 reason = Timeout
55712025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
55722025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
55732025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 15 reason = Timeout
55742025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
55752025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 8 reason = Timeout
55762025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
55772025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 8 reason = Timeout
55782025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
55792025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 15 reason = Timeout
55802025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
55812025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 15 reason = Timeout
55822025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55832025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
55842025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
55852025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
55862025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
55872025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
55882025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
55892025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
55902025-02-28T12:06:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
55912025-02-28T12:06:10.461ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
55922025-02-28T12:06:10.461ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
55932025-02-28T12:06:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
55942025-02-28T12:06:10.461ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.630532ms iteration = 15 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
55952025-02-28T12:06:10.463ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55962025-02-28T12:06:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
55972025-02-28T12:06:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
55982025-02-28T12:06:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
55992025-02-28T12:06:10.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56002025-02-28T12:06:10.464ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
56012025-02-28T12:06:10.464ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
56022025-02-28T12:06:10.464ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
56032025-02-28T12:06:10.464ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
56042025-02-28T12:06:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56052025-02-28T12:06:10.464ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56062025-02-28T12:06:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
56072025-02-28T12:06:10.465ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
56082025-02-28T12:06:10.465ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
56092025-02-28T12:06:10.465ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
56102025-02-28T12:06:10.465ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
56112025-02-28T12:06:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
56122025-02-28T12:06:10.465ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
56132025-02-28T12:06:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
56142025-02-28T12:06:10.465ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56152025-02-28T12:06:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
56162025-02-28T12:06:10.466ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56172025-02-28T12:06:10.466ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
56182025-02-28T12:06:10.466ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56192025-02-28T12:06:10.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56202025-02-28T12:06:10.467ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56212025-02-28T12:06:10.467ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
56222025-02-28T12:06:10.468ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
56232025-02-28T12:06:10.468ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
56242025-02-28T12:06:10.468ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
56252025-02-28T12:06:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
56262025-02-28T12:06:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
56272025-02-28T12:06:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
56282025-02-28T12:06:10.468ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
56292025-02-28T12:06:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
56302025-02-28T12:06:10.468ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56312025-02-28T12:06:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
56322025-02-28T12:06:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56332025-02-28T12:06:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
56342025-02-28T12:06:10.469ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56352025-02-28T12:06:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56362025-02-28T12:06:10.470ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56372025-02-28T12:06:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56382025-02-28T12:06:10.470ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56392025-02-28T12:06:10.471ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56402025-02-28T12:06:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
56412025-02-28T12:06:10.472ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56422025-02-28T12:06:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56432025-02-28T12:06:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
56442025-02-28T12:06:10.473ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
56452025-02-28T12:06:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
56462025-02-28T12:06:10.474ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 13.849344ms iteration = 8 status = {"errors":[],"found_port_ok":[]}
56472025-02-28T12:06:10.474ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 14.056759ms iteration = 15 status = {"errors":[],"requests_created_ok":[]}
56482025-02-28T12:06:10.474ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 14.594583ms iteration = 15 status = {"errors":[],"finish_invoked_ok":[]}
56492025-02-28T12:06:10.475ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 14.673744ms iteration = 15 status = {"errors":[],"garbage_collect_requested":[]}
56502025-02-28T12:06:10.475ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
56512025-02-28T12:06:10.475ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
56522025-02-28T12:06:10.475ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 14.797227ms iteration = 8 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
56532025-02-28T12:06:10.476ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
56542025-02-28T12:06:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
56552025-02-28T12:06:10.476ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56562025-02-28T12:06:10.476ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56572025-02-28T12:06:10.476ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
56582025-02-28T12:06:10.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56592025-02-28T12:06:10.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
56602025-02-28T12:06:10.477ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
56612025-02-28T12:06:10.477ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56622025-02-28T12:06:10.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
56632025-02-28T12:06:10.478ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56642025-02-28T12:06:10.479ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
56652025-02-28T12:06:10.479ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
56662025-02-28T12:06:10.479ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
56672025-02-28T12:06:10.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
56682025-02-28T12:06:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
56692025-02-28T12:06:10.480ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
56702025-02-28T12:06:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
56712025-02-28T12:06:10.480ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
56722025-02-28T12:06:10.480ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56732025-02-28T12:06:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56742025-02-28T12:06:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
56752025-02-28T12:06:10.481ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
56762025-02-28T12:06:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
56772025-02-28T12:06:10.483ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56782025-02-28T12:06:10.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56792025-02-28T12:06:10.484ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56802025-02-28T12:06:10.484ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1205, tv_nsec: 120366088 } })
56812025-02-28T12:06:10.484ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56822025-02-28T12:06:10.485ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
56832025-02-28T12:06:10.485ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56842025-02-28T12:06:10.485ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56852025-02-28T12:06:10.488ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
56862025-02-28T12:06:10.488ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56872025-02-28T12:06:10.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56882025-02-28T12:06:10.489ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
56892025-02-28T12:06:10.489ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
56902025-02-28T12:06:10.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
56912025-02-28T12:06:10.492ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56922025-02-28T12:06:10.494ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
56932025-02-28T12:06:10.494ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56942025-02-28T12:06:10.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56952025-02-28T12:06:10.513ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
56962025-02-28T12:06:10.513ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 56.484554ms iteration = 43 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
56972025-02-28T12:06:10.533ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
56982025-02-28T12:06:10.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
56992025-02-28T12:06:10.536ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 76.842366ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
57002025-02-28T12:06:10.537ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
57012025-02-28T12:06:10.537ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
57022025-02-28T12:06:10.537ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 78.351382ms iteration = 8 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
57032025-02-28T12:06:10.542ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 83.25805ms iteration = 43 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
57042025-02-28T12:06:10.542ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57052025-02-28T12:06:10.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57062025-02-28T12:06:10.544ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57072025-02-28T12:06:10.544ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57082025-02-28T12:06:10.546ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
57092025-02-28T12:06:10.546ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 87.656966ms iteration = 15 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
57102025-02-28T12:06:10.546ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57112025-02-28T12:06:10.547ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57122025-02-28T12:06:10.547ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57132025-02-28T12:06:10.549ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57142025-02-28T12:06:10.549ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
57152025-02-28T12:06:10.549ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
57162025-02-28T12:06:10.549ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
57172025-02-28T12:06:10.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
57182025-02-28T12:06:10.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
57192025-02-28T12:06:10.550ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
57202025-02-28T12:06:10.550ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
57212025-02-28T12:06:10.550ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 91.035778ms iteration = 15 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
57222025-02-28T12:06:10.550ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
57232025-02-28T12:06:10.550ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
57242025-02-28T12:06:10.550ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
57252025-02-28T12:06:10.551ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
57262025-02-28T12:06:10.552ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 92.079763ms iteration = 15 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
57272025-02-28T12:06:10.552ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
57282025-02-28T12:06:10.552ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57292025-02-28T12:06:10.552ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57302025-02-28T12:06:10.552ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57312025-02-28T12:06:10.554ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
57322025-02-28T12:06:10.554ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 0 version = 2
57332025-02-28T12:06:10.554ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
57342025-02-28T12:06:10.554ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 95.574307ms iteration = 15 status = 0
57352025-02-28T12:06:10.554ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
57362025-02-28T12:06:10.555ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
57372025-02-28T12:06:10.555ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
57382025-02-28T12:06:10.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
57392025-02-28T12:06:10.558ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
57402025-02-28T12:06:10.558ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
57412025-02-28T12:06:10.559ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
57422025-02-28T12:06:10.560ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
57432025-02-28T12:06:10.560ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-02-28 11:59:51.915187 UTC dns_group = internal generation = 1
57442025-02-28T12:06:10.560ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 105.975098ms iteration = 8 status = {"generation":1}
57452025-02-28T12:06:10.561ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
57462025-02-28T12:06:10.561ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
57472025-02-28T12:06:10.563ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57482025-02-28T12:06:10.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57492025-02-28T12:06:10.565ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57502025-02-28T12:06:10.567ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
57512025-02-28T12:06:10.567ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57522025-02-28T12:06:10.567ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57532025-02-28T12:06:10.569ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57542025-02-28T12:06:10.571ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
57552025-02-28T12:06:10.571ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57562025-02-28T12:06:10.571ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57572025-02-28T12:06:10.574ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 114.111885ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
57582025-02-28T12:06:10.589ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
57592025-02-28T12:06:10.589ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 130.662394ms iteration = 15 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
57602025-02-28T12:06:10.591ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
57612025-02-28T12:06:10.592ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57622025-02-28T12:06:10.595ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57632025-02-28T12:06:10.596ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
57642025-02-28T12:06:10.596ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
57652025-02-28T12:06:10.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
57662025-02-28T12:06:10.598ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
57672025-02-28T12:06:10.598ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
57682025-02-28T12:06:10.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
57692025-02-28T12:06:10.609ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
57702025-02-28T12:06:10.611ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
57712025-02-28T12:06:10.611ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
57722025-02-28T12:06:10.611ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d590e212-edf1-41f1-b784-a4522844f73b", "content-length": "26", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} })
57732025-02-28T12:06:10.611ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
57742025-02-28T12:06:10.615ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
57752025-02-28T12:06:10.646ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
57762025-02-28T12:06:10.646ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
57772025-02-28T12:06:10.649ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "41011793-d672-4e76-9a9e-9bdcb7151929", "content-length": "26", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} })
57782025-02-28T12:06:10.649ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
57792025-02-28T12:06:10.652ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
57802025-02-28T12:06:10.657ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
57812025-02-28T12:06:10.658ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 198.663816ms iteration = 8 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
57822025-02-28T12:06:10.667ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 207.788456ms iteration = 16 status = {"physical_disks_added":0}
57832025-02-28T12:06:10.683ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
57842025-02-28T12:06:10.683ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
57852025-02-28T12:06:10.683ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "21e1733a-cd9f-429f-ad76-3e75101d1b16", "content-length": "26", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} })
57862025-02-28T12:06:10.683ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
57872025-02-28T12:06:10.686ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
57882025-02-28T12:06:10.825ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
57892025-02-28T12:06:10.826ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4be3ce1-06f8-45d3-b019-4c3329cdd60e", "content-length": "691", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} })
57902025-02-28T12:06:10.826ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
57912025-02-28T12:06:10.826ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
57922025-02-28T12:06:10.826ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
57932025-02-28T12:06:10.828ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 368.890455ms iteration = 15 status = {}
57942025-02-28T12:06:10.828ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
57952025-02-28T12:06:10.828ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
57962025-02-28T12:06:10.829ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9d443bc-d66c-4842-8abf-23d69982b3b2", "content-length": "372", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} })
57972025-02-28T12:06:10.829ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57982025-02-28T12:06:10.829ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57992025-02-28T12:06:10.830ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 1 version = 2
58002025-02-28T12:06:10.830ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
58012025-02-28T12:06:10.830ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-02-28 11:59:52.542241 UTC dns_group = external generation = 2
58022025-02-28T12:06:10.830ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 376.750884ms iteration = 8 status = {"generation":2}
58032025-02-28T12:06:10.835ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
58042025-02-28T12:06:10.836ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "6f1bcdbb-451a-4956-a5fb-3cc5d90d3522", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
58052025-02-28T12:06:10.836ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
58062025-02-28T12:06:10.836ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
58072025-02-28T12:06:10.837ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "395cc54f-36cb-4344-8826-c5c670645b93", "content-length": "1", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} })
58082025-02-28T12:06:10.838ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "6d829422-4949-4fa7-b063-ab30a3306254", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
58092025-02-28T12:06:10.838ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
58102025-02-28T12:06:10.838ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
58112025-02-28T12:06:10.838ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
58122025-02-28T12:06:10.840ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
58132025-02-28T12:06:10.840ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 386.189973ms iteration = 10 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
58142025-02-28T12:06:10.841ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 386.974922ms iteration = 15 status = null
58152025-02-28T12:06:10.849ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 395.468817ms iteration = 15 status = {}
58162025-02-28T12:06:10.851ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
58172025-02-28T12:06:10.851ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
58182025-02-28T12:06:10.851ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "30ed062d-9a49-4d29-a3a2-65a8fc426d79", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} })
58192025-02-28T12:06:10.851ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58202025-02-28T12:06:10.851ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58212025-02-28T12:06:10.855ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58222025-02-28T12:06:10.855ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58232025-02-28T12:06:10.859ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58242025-02-28T12:06:10.859ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58252025-02-28T12:06:10.863ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
58262025-02-28T12:06:10.863ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
58272025-02-28T12:06:10.863ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "cd7191c5-f1a7-4358-a9bf-77929b75bb7f", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} })
58282025-02-28T12:06:10.865ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 406.755469ms iteration = 15 status = {}
58292025-02-28T12:06:10.880ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
58302025-02-28T12:06:10.884ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-28T11:59:08.719399Z, time_modified: 2025-02-28T11:59:08.719399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }) }
58312025-02-28T12:06:10.884ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 430.702578ms iteration = 8 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]}},"default_endpoint":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
58322025-02-28T12:06:10.993ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
58332025-02-28T12:06:10.994ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "9fef43c3-7da7-4a1b-9e13-f831b8664d83", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
58342025-02-28T12:06:10.994ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
58352025-02-28T12:06:10.995ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "5fee5ff8-480c-4caa-96cb-993c47c18659", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
58362025-02-28T12:06:10.995ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
58372025-02-28T12:06:10.998ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "82650b2a-ad1c-4f17-976d-3edec2a584be", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
58382025-02-28T12:06:10.998ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
58392025-02-28T12:06:11.000ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
58402025-02-28T12:06:11.000ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 546.036352ms iteration = 10 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
58412025-02-28T12:06:11.058ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58422025-02-28T12:06:11.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58432025-02-28T12:06:11.060ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
58442025-02-28T12:06:11.060ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58452025-02-28T12:06:11.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58462025-02-28T12:06:11.070ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
58472025-02-28T12:06:11.071ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8debfa1-c5e0-4e16-9c08-2df2c955efda", "content-length": "134", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} })
58482025-02-28T12:06:11.071ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
58492025-02-28T12:06:11.071ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
58502025-02-28T12:06:11.071ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
58512025-02-28T12:06:11.071ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
58522025-02-28T12:06:11.071ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58532025-02-28T12:06:11.071ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58542025-02-28T12:06:11.071ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58552025-02-28T12:06:11.071ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
58562025-02-28T12:06:11.071ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58572025-02-28T12:06:11.071ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
58582025-02-28T12:06:11.071ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
58592025-02-28T12:06:11.071ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58602025-02-28T12:06:11.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58612025-02-28T12:06:11.073ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
58622025-02-28T12:06:11.074ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "9adcbf65-0a76-4e6d-9fba-91d79b38ab51", "content-length": "2", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} })
58632025-02-28T12:06:11.074ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
58642025-02-28T12:06:11.074ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d219485-5059-4a0f-b412-f7d9930afe36", "content-length": "51", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} })
58652025-02-28T12:06:11.074ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
58662025-02-28T12:06:11.074ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
58672025-02-28T12:06:11.177ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "d858970e-6785-40fd-be09-6da4590df2d3", "date": "Fri, 28 Feb 2025 12:06:10 GMT"} })
58682025-02-28T12:06:11.177ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
58692025-02-28T12:06:11.177ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
58702025-02-28T12:06:11.178ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ce753f0-f6cf-492d-9bac-5b840fff18b4", "content-length": "682", "date": "Fri, 28 Feb 2025 12:06:11 GMT"} })
58712025-02-28T12:06:11.178ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58722025-02-28T12:06:11.178ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58732025-02-28T12:06:11.183ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58742025-02-28T12:06:11.183ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58752025-02-28T12:06:11.186ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58762025-02-28T12:06:11.187ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58772025-02-28T12:06:11.189ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58782025-02-28T12:06:11.189ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58792025-02-28T12:06:11.195ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 736.738996ms iteration = 15 status = {}
58802025-02-28T12:06:14.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 289 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = b602f51d-56f4-4c33-a046-6d8b7f57bdfc response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
58812025-02-28T12:06:20.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
58822025-02-28T12:06:20.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58832025-02-28T12:06:20.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58842025-02-28T12:06:20.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 44 reason = Timeout
58852025-02-28T12:06:20.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58862025-02-28T12:06:20.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58872025-02-28T12:06:20.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
58882025-02-28T12:06:20.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
58892025-02-28T12:06:20.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58902025-02-28T12:06:20.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58912025-02-28T12:06:20.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58922025-02-28T12:06:20.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58932025-02-28T12:06:20.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58942025-02-28T12:06:20.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
58952025-02-28T12:06:20.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58962025-02-28T12:06:20.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58972025-02-28T12:06:20.485ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
58982025-02-28T12:06:20.485ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 27.849152ms iteration = 44 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
58992025-02-28T12:06:20.491ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59002025-02-28T12:06:20.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59012025-02-28T12:06:20.523ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.413555ms iteration = 44 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
59022025-02-28T12:06:24.254ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 015a5fa3-4808-49e9-9111-b3ba5f024ae4 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
59032025-02-28T12:06:24.254ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 015a5fa3-4808-49e9-9111-b3ba5f024ae4 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
59042025-02-28T12:06:24.256ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2873 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 015a5fa3-4808-49e9-9111-b3ba5f024ae4 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
59052025-02-28T12:06:24.257ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 9a53c569-f5e8-493b-b284-95a1f78f090f roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
59062025-02-28T12:06:24.257ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 9a53c569-f5e8-493b-b284-95a1f78f090f resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
59072025-02-28T12:06:24.259ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2397 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 9a53c569-f5e8-493b-b284-95a1f78f090f response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
59082025-02-28T12:06:24.262ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 266 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 4563b0c6-39c2-4648-9854-d914e492b1f5 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
59092025-02-28T12:06:30.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
59102025-02-28T12:06:30.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59112025-02-28T12:06:30.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59122025-02-28T12:06:30.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 45 reason = Timeout
59132025-02-28T12:06:30.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59142025-02-28T12:06:30.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59152025-02-28T12:06:30.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59162025-02-28T12:06:30.461ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59172025-02-28T12:06:30.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59182025-02-28T12:06:30.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
59192025-02-28T12:06:30.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59202025-02-28T12:06:30.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59212025-02-28T12:06:30.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59222025-02-28T12:06:30.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59232025-02-28T12:06:30.464ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.414627ms iteration = 45 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
59242025-02-28T12:06:30.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59252025-02-28T12:06:30.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
59262025-02-28T12:06:30.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59272025-02-28T12:06:30.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59282025-02-28T12:06:30.477ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
59292025-02-28T12:06:30.477ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.368352ms iteration = 45 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
59302025-02-28T12:06:34.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 261 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 32e762ed-691c-4064-be0b-a5f0d794f42e response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
59312025-02-28T12:06:40.453ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
59322025-02-28T12:06:40.454ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
59332025-02-28T12:06:40.454ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
59342025-02-28T12:06:40.454ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
59352025-02-28T12:06:40.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
59362025-02-28T12:06:40.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
59372025-02-28T12:06:40.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
59382025-02-28T12:06:40.456ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
59392025-02-28T12:06:40.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59402025-02-28T12:06:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59412025-02-28T12:06:40.457ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1235, tv_nsec: 93388427 } })
59422025-02-28T12:06:40.457ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
59432025-02-28T12:06:40.457ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1235, tv_nsec: 93609492 } })
59442025-02-28T12:06:40.458ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
59452025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59462025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 46 reason = Timeout
59472025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59482025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 16 reason = Timeout
59492025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 16 reason = Timeout
59502025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
59512025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
59522025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
59532025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 16 reason = Timeout
59542025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 16 reason = Timeout
59552025-02-28T12:06:40.459ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
59562025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
59572025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59582025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
59592025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
59602025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 17 reason = Timeout
59612025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 16 reason = Timeout
59622025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
59632025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
59642025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
59652025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 16 reason = Timeout
59662025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
59672025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
59682025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59692025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
59702025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.543732ms iteration = 16 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
59712025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
59722025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
59732025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
59742025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
59752025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
59762025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
59772025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
59782025-02-28T12:06:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
59792025-02-28T12:06:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 16 reason = Timeout
59802025-02-28T12:06:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 16 reason = Timeout
59812025-02-28T12:06:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
59822025-02-28T12:06:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
59832025-02-28T12:06:40.460ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 16 reason = Timeout
59842025-02-28T12:06:40.460ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
59852025-02-28T12:06:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 16 reason = Timeout
59862025-02-28T12:06:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
59872025-02-28T12:06:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 16 reason = Timeout
59882025-02-28T12:06:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
59892025-02-28T12:06:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
59902025-02-28T12:06:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
59912025-02-28T12:06:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
59922025-02-28T12:06:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
59932025-02-28T12:06:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
59942025-02-28T12:06:40.461ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59952025-02-28T12:06:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59962025-02-28T12:06:40.461ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59972025-02-28T12:06:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
59982025-02-28T12:06:40.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59992025-02-28T12:06:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60002025-02-28T12:06:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
60012025-02-28T12:06:40.463ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
60022025-02-28T12:06:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
60032025-02-28T12:06:40.463ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60042025-02-28T12:06:40.464ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60052025-02-28T12:06:40.464ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60062025-02-28T12:06:40.465ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60072025-02-28T12:06:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
60082025-02-28T12:06:40.465ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
60092025-02-28T12:06:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
60102025-02-28T12:06:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
60112025-02-28T12:06:40.466ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
60122025-02-28T12:06:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
60132025-02-28T12:06:40.466ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
60142025-02-28T12:06:40.466ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
60152025-02-28T12:06:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60162025-02-28T12:06:40.467ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60172025-02-28T12:06:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
60182025-02-28T12:06:40.467ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
60192025-02-28T12:06:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60202025-02-28T12:06:40.467ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 7.412479ms iteration = 16 status = {"errors":[],"finish_invoked_ok":[]}
60212025-02-28T12:06:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
60222025-02-28T12:06:40.469ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60232025-02-28T12:06:40.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60242025-02-28T12:06:40.471ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 12.899612ms iteration = 46 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
60252025-02-28T12:06:40.472ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60262025-02-28T12:06:40.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60272025-02-28T12:06:40.473ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
60282025-02-28T12:06:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
60292025-02-28T12:06:40.473ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1235, tv_nsec: 109710691 } })
60302025-02-28T12:06:40.474ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
60312025-02-28T12:06:40.475ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
60322025-02-28T12:06:40.475ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60332025-02-28T12:06:40.476ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60342025-02-28T12:06:40.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60352025-02-28T12:06:40.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60362025-02-28T12:06:40.478ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
60372025-02-28T12:06:40.478ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60382025-02-28T12:06:40.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60392025-02-28T12:06:40.493ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
60402025-02-28T12:06:40.493ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
60412025-02-28T12:06:40.493ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
60422025-02-28T12:06:40.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
60432025-02-28T12:06:40.495ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
60442025-02-28T12:06:40.495ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 38.804697ms iteration = 46 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
60452025-02-28T12:06:40.496ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
60462025-02-28T12:06:40.496ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
60472025-02-28T12:06:40.496ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
60482025-02-28T12:06:40.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
60492025-02-28T12:06:40.500ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
60502025-02-28T12:06:40.500ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
60512025-02-28T12:06:40.500ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 41.202784ms iteration = 16 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
60522025-02-28T12:06:40.505ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60532025-02-28T12:06:40.505ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60542025-02-28T12:06:40.507ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 46.669407ms iteration = 16 status = {"errors":[],"requests_created_ok":[]}
60552025-02-28T12:06:40.508ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60562025-02-28T12:06:40.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60572025-02-28T12:06:40.509ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
60582025-02-28T12:06:40.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
60592025-02-28T12:06:40.509ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 49.318051ms iteration = 16 status = {"errors":[],"garbage_collect_requested":[]}
60602025-02-28T12:06:40.510ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60612025-02-28T12:06:40.511ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
60622025-02-28T12:06:40.511ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60632025-02-28T12:06:40.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
60642025-02-28T12:06:40.513ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
60652025-02-28T12:06:40.513ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
60662025-02-28T12:06:40.513ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60672025-02-28T12:06:40.513ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
60682025-02-28T12:06:40.513ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 54.932426ms iteration = 16 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
60692025-02-28T12:06:40.514ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
60702025-02-28T12:06:40.514ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
60712025-02-28T12:06:40.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60722025-02-28T12:06:40.515ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
60732025-02-28T12:06:40.515ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
60742025-02-28T12:06:40.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
60752025-02-28T12:06:40.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
60762025-02-28T12:06:40.515ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 55.391167ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
60772025-02-28T12:06:40.517ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 58.727948ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
60782025-02-28T12:06:40.519ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
60792025-02-28T12:06:40.519ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
60802025-02-28T12:06:40.519ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
60812025-02-28T12:06:40.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
60822025-02-28T12:06:40.519ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "80de92fe-ca87-4892-91fa-bff0aefb8d1b", "content-length": "26", "date": "Fri, 28 Feb 2025 12:06:40 GMT"} })
60832025-02-28T12:06:40.519ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
60842025-02-28T12:06:40.522ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
60852025-02-28T12:06:40.526ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
60862025-02-28T12:06:40.526ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
60872025-02-28T12:06:40.526ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bba03231-2280-4eb8-a454-bec741735db0", "content-length": "26", "date": "Fri, 28 Feb 2025 12:06:40 GMT"} })
60882025-02-28T12:06:40.526ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
60892025-02-28T12:06:40.531ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
60902025-02-28T12:06:40.533ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
60912025-02-28T12:06:40.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
60922025-02-28T12:06:40.536ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60932025-02-28T12:06:40.538ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
60942025-02-28T12:06:40.538ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
60952025-02-28T12:06:40.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
60962025-02-28T12:06:40.540ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60972025-02-28T12:06:40.542ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
60982025-02-28T12:06:40.542ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
60992025-02-28T12:06:40.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
61002025-02-28T12:06:40.550ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
61012025-02-28T12:06:40.550ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
61022025-02-28T12:06:40.551ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b14a27a-1a7f-4a37-a629-73b5ffbf3f67", "content-length": "26", "date": "Fri, 28 Feb 2025 12:06:40 GMT"} })
61032025-02-28T12:06:40.551ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
61042025-02-28T12:06:40.553ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
61052025-02-28T12:06:40.553ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 93.095427ms iteration = 16 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
61062025-02-28T12:06:40.559ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61072025-02-28T12:06:40.562ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
61082025-02-28T12:06:40.563ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
61092025-02-28T12:06:40.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
61102025-02-28T12:06:40.564ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61112025-02-28T12:06:40.566ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 107.035884ms iteration = 16 status = 0
61122025-02-28T12:06:40.566ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
61132025-02-28T12:06:40.567ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
61142025-02-28T12:06:40.567ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
61152025-02-28T12:06:40.604ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
61162025-02-28T12:06:40.604ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "9467aa96-2a22-4c1a-92b5-5615e16710b6", "content-length": "1", "date": "Fri, 28 Feb 2025 12:06:40 GMT"} })
61172025-02-28T12:06:40.606ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
61182025-02-28T12:06:40.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
61192025-02-28T12:06:40.609ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 155.838112ms iteration = 16 status = null
61202025-02-28T12:06:40.614ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
61212025-02-28T12:06:40.615ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 155.906253ms iteration = 16 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
61222025-02-28T12:06:40.624ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 170.806372ms iteration = 16 status = {}
61232025-02-28T12:06:40.656ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 197.304023ms iteration = 17 status = {"physical_disks_added":0}
61242025-02-28T12:06:40.705ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
61252025-02-28T12:06:40.705ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
61262025-02-28T12:06:40.705ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "22838242-24b8-4500-86c7-3d71b1a5b375", "content-length": "372", "date": "Fri, 28 Feb 2025 12:06:40 GMT"} })
61272025-02-28T12:06:40.705ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61282025-02-28T12:06:40.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61292025-02-28T12:06:40.730ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
61302025-02-28T12:06:40.731ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "f138baed-c164-4de5-aece-6b0fa904a86e", "content-length": "691", "date": "Fri, 28 Feb 2025 12:06:40 GMT"} })
61312025-02-28T12:06:40.731ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
61322025-02-28T12:06:40.731ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
61332025-02-28T12:06:40.731ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
61342025-02-28T12:06:40.734ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 274.126566ms iteration = 16 status = {}
61352025-02-28T12:06:40.761ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
61362025-02-28T12:06:40.761ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
61372025-02-28T12:06:40.761ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "9e2c8c67-cda5-40d9-8074-5946b5368a87", "date": "Fri, 28 Feb 2025 12:06:40 GMT"} })
61382025-02-28T12:06:40.761ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61392025-02-28T12:06:40.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61402025-02-28T12:06:40.769ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61412025-02-28T12:06:40.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61422025-02-28T12:06:40.773ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61432025-02-28T12:06:40.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61442025-02-28T12:06:40.778ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
61452025-02-28T12:06:40.778ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
61462025-02-28T12:06:40.778ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "456200f6-27e2-456e-9f5c-eeed6d120b28", "date": "Fri, 28 Feb 2025 12:06:40 GMT"} })
61472025-02-28T12:06:40.780ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 321.160333ms iteration = 16 status = {}
61482025-02-28T12:06:40.790ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61492025-02-28T12:06:40.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61502025-02-28T12:06:40.792ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
61512025-02-28T12:06:40.792ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61522025-02-28T12:06:40.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61532025-02-28T12:06:40.805ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
61542025-02-28T12:06:40.806ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "860cb937-1d11-477b-92a5-2ef3b14da857", "content-length": "134", "date": "Fri, 28 Feb 2025 12:06:40 GMT"} })
61552025-02-28T12:06:40.806ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
61562025-02-28T12:06:40.806ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
61572025-02-28T12:06:40.806ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
61582025-02-28T12:06:40.806ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
61592025-02-28T12:06:40.806ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
61602025-02-28T12:06:40.806ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
61612025-02-28T12:06:40.806ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
61622025-02-28T12:06:40.806ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
61632025-02-28T12:06:40.807ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
61642025-02-28T12:06:40.807ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
61652025-02-28T12:06:40.807ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
61662025-02-28T12:06:40.807ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61672025-02-28T12:06:40.807ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61682025-02-28T12:06:40.809ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
61692025-02-28T12:06:40.809ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "df060443-fb27-4ba7-bc73-73b9de2480e7", "content-length": "2", "date": "Fri, 28 Feb 2025 12:06:40 GMT"} })
61702025-02-28T12:06:40.809ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
61712025-02-28T12:06:40.810ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "818c3609-e6fa-4231-9223-2f30ac44a63b", "content-length": "51", "date": "Fri, 28 Feb 2025 12:06:40 GMT"} })
61722025-02-28T12:06:40.810ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
61732025-02-28T12:06:40.810ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
61742025-02-28T12:06:40.935ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "b6652cb2-2f85-4472-a8f2-5321194502f7", "date": "Fri, 28 Feb 2025 12:06:40 GMT"} })
61752025-02-28T12:06:40.935ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
61762025-02-28T12:06:40.935ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
61772025-02-28T12:06:40.936ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "41eddebd-b3c3-4a1f-bf3f-7c4ee1e2acb1", "content-length": "682", "date": "Fri, 28 Feb 2025 12:06:40 GMT"} })
61782025-02-28T12:06:40.936ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61792025-02-28T12:06:40.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61802025-02-28T12:06:40.940ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61812025-02-28T12:06:40.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61822025-02-28T12:06:40.943ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61832025-02-28T12:06:40.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61842025-02-28T12:06:40.946ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61852025-02-28T12:06:40.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61862025-02-28T12:06:40.954ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 495.593164ms iteration = 16 status = {}
61872025-02-28T12:06:44.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 296 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 7f6fb778-78ef-4b57-8c40-dd46492e1c57 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
61882025-02-28T12:06:50.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
61892025-02-28T12:06:50.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61902025-02-28T12:06:50.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61912025-02-28T12:06:50.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 47 reason = Timeout
61922025-02-28T12:06:50.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
61932025-02-28T12:06:50.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
61942025-02-28T12:06:50.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61952025-02-28T12:06:50.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
61962025-02-28T12:06:50.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
61972025-02-28T12:06:50.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
61982025-02-28T12:06:50.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61992025-02-28T12:06:50.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62002025-02-28T12:06:50.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62012025-02-28T12:06:50.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62022025-02-28T12:06:50.469ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62032025-02-28T12:06:50.471ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
62042025-02-28T12:06:50.471ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62052025-02-28T12:06:50.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62062025-02-28T12:06:50.486ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
62072025-02-28T12:06:50.486ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 28.714324ms iteration = 47 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
62082025-02-28T12:06:50.496ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 37.547127ms iteration = 47 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
62092025-02-28T12:06:54.254ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = fc666ee1-aa08-4595-b61c-994b7a93d532 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
62102025-02-28T12:06:54.254ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = fc666ee1-aa08-4595-b61c-994b7a93d532 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
62112025-02-28T12:06:54.257ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3114 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = fc666ee1-aa08-4595-b61c-994b7a93d532 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
62122025-02-28T12:06:54.257ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 0580df57-3425-4e9f-91f9-99d3349a4e09 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
62132025-02-28T12:06:54.258ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 0580df57-3425-4e9f-91f9-99d3349a4e09 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
62142025-02-28T12:06:54.260ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2523 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 0580df57-3425-4e9f-91f9-99d3349a4e09 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
62152025-02-28T12:06:54.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 283 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 0dfb1e36-68a4-46f8-acd1-e428945d34c6 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
62162025-02-28T12:07:00.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
62172025-02-28T12:07:00.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62182025-02-28T12:07:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62192025-02-28T12:07:00.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 48 reason = Timeout
62202025-02-28T12:07:00.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62212025-02-28T12:07:00.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62222025-02-28T12:07:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62232025-02-28T12:07:00.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62242025-02-28T12:07:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62252025-02-28T12:07:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
62262025-02-28T12:07:00.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62272025-02-28T12:07:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62282025-02-28T12:07:00.462ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.948206ms iteration = 48 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
62292025-02-28T12:07:00.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62302025-02-28T12:07:00.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62312025-02-28T12:07:00.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62322025-02-28T12:07:00.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
62332025-02-28T12:07:00.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62342025-02-28T12:07:00.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62352025-02-28T12:07:00.479ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
62362025-02-28T12:07:00.479ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.441292ms iteration = 48 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
62372025-02-28T12:07:04.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 326 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = b16a63a5-7672-497f-af31-b699d4b81492 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
62382025-02-28T12:07:09.254ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = d6c00bb7-883f-49cd-88bb-310ca6b60987 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
62392025-02-28T12:07:09.254ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = d6c00bb7-883f-49cd-88bb-310ca6b60987 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
62402025-02-28T12:07:09.257ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3208 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = d6c00bb7-883f-49cd-88bb-310ca6b60987 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
62412025-02-28T12:07:09.257ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 5d204578-610d-4d68-b582-4cfb49c37f22 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
62422025-02-28T12:07:09.257ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 5d204578-610d-4d68-b582-4cfb49c37f22 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
62432025-02-28T12:07:09.259ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2067 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 5d204578-610d-4d68-b582-4cfb49c37f22 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
62442025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 9 reason = Timeout
62452025-02-28T12:07:10.454ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-28 11:57:10.454670771 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
62462025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
62472025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
62482025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
62492025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
62502025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
62512025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
62522025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
62532025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
62542025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
62552025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
62562025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
62572025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
62582025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
62592025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
62602025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
62612025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
62622025-02-28T12:07:10.454ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
62632025-02-28T12:07:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
62642025-02-28T12:07:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
62652025-02-28T12:07:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
62662025-02-28T12:07:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
62672025-02-28T12:07:10.455ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5cefabf1-6831-46ff-98eb-1a3b5abad8b9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1265, tv_nsec: 91287596 } })
62682025-02-28T12:07:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
62692025-02-28T12:07:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
62702025-02-28T12:07:10.455ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5ebff2d9-48b6-4fe6-b852-684bb14b4955.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bddfb0f9-1d7b-45e4-b3a5-35aa6792956b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1265, tv_nsec: 91349967 } })
62712025-02-28T12:07:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
62722025-02-28T12:07:10.455ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
62732025-02-28T12:07:10.455ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.089536ms iteration = 9 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
62742025-02-28T12:07:10.455ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
62752025-02-28T12:07:10.455ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.126297ms iteration = 9 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
62762025-02-28T12:07:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
62772025-02-28T12:07:10.457ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
62782025-02-28T12:07:10.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
62792025-02-28T12:07:10.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
62802025-02-28T12:07:10.457ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
62812025-02-28T12:07:10.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62822025-02-28T12:07:10.457ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 8.59\u{b5}s iteration = 10 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
62832025-02-28T12:07:10.457ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
62842025-02-28T12:07:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
62852025-02-28T12:07:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62862025-02-28T12:07:10.457ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-28 11:57:10.454670771 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
62872025-02-28T12:07:10.457ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.034434ms iteration = 9 status = {"expiration":"2025-02-28T11:57:10.454670771Z","pruned":[]}
62882025-02-28T12:07:10.458ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1265, tv_nsec: 94170286 } })
62892025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62902025-02-28T12:07:10.458ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
62912025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 49 reason = Timeout
62922025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 17 reason = Timeout
62932025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62942025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
62952025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 17 reason = Timeout
62962025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
62972025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
62982025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 9 reason = Timeout
62992025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
63002025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 17 reason = Timeout
63012025-02-28T12:07:10.458ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
63022025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 18 reason = Timeout
63032025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63042025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 17 reason = Timeout
63052025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
63062025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
63072025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 17 reason = Timeout
63082025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63092025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
63102025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
63112025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 9 reason = Timeout
63122025-02-28T12:07:10.458ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
63132025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
63142025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
63152025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
63162025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 17 reason = Timeout
63172025-02-28T12:07:10.458ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
63182025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
63192025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63202025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63212025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
63222025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
63232025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
63242025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
63252025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63262025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
63272025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
63282025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
63292025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
63302025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63312025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 17 reason = Timeout
63322025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
63332025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 9 reason = Timeout
63342025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 17 reason = Timeout
63352025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 9 reason = Timeout
63362025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
63372025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
63382025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
63392025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 17 reason = Timeout
63402025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
63412025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 17 reason = Timeout
63422025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
63432025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 17 reason = Timeout
63442025-02-28T12:07:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
63452025-02-28T12:07:10.459ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1265, tv_nsec: 95955449 } })
63462025-02-28T12:07:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
63472025-02-28T12:07:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
63482025-02-28T12:07:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
63492025-02-28T12:07:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
63502025-02-28T12:07:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
63512025-02-28T12:07:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
63522025-02-28T12:07:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
63532025-02-28T12:07:10.460ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
63542025-02-28T12:07:10.460ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63552025-02-28T12:07:10.460ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
63562025-02-28T12:07:10.460ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
63572025-02-28T12:07:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
63582025-02-28T12:07:10.461ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.297644ms iteration = 17 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
63592025-02-28T12:07:10.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63602025-02-28T12:07:10.462ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63612025-02-28T12:07:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
63622025-02-28T12:07:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
63632025-02-28T12:07:10.464ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
63642025-02-28T12:07:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63652025-02-28T12:07:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
63662025-02-28T12:07:10.465ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
63672025-02-28T12:07:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
63682025-02-28T12:07:10.465ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63692025-02-28T12:07:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
63702025-02-28T12:07:10.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63712025-02-28T12:07:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63722025-02-28T12:07:10.466ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
63732025-02-28T12:07:10.466ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
63742025-02-28T12:07:10.466ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
63752025-02-28T12:07:10.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
63762025-02-28T12:07:10.467ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
63772025-02-28T12:07:10.467ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63782025-02-28T12:07:10.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63792025-02-28T12:07:10.468ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63802025-02-28T12:07:10.468ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63812025-02-28T12:07:10.469ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63822025-02-28T12:07:10.470ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63832025-02-28T12:07:10.470ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63842025-02-28T12:07:10.470ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
63852025-02-28T12:07:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
63862025-02-28T12:07:10.471ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 11.360024ms iteration = 9 status = {"errors":[],"found_port_ok":[]}
63872025-02-28T12:07:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
63882025-02-28T12:07:10.471ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63892025-02-28T12:07:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63902025-02-28T12:07:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
63912025-02-28T12:07:10.472ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
63922025-02-28T12:07:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
63932025-02-28T12:07:10.472ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
63942025-02-28T12:07:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
63952025-02-28T12:07:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
63962025-02-28T12:07:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
63972025-02-28T12:07:10.473ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
63982025-02-28T12:07:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
63992025-02-28T12:07:10.473ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
64002025-02-28T12:07:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
64012025-02-28T12:07:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
64022025-02-28T12:07:10.477ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 19.325527ms iteration = 49 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
64032025-02-28T12:07:10.478ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
64042025-02-28T12:07:10.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
64052025-02-28T12:07:10.478ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64062025-02-28T12:07:10.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64072025-02-28T12:07:10.479ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64082025-02-28T12:07:10.480ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
64092025-02-28T12:07:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
64102025-02-28T12:07:10.480ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
64112025-02-28T12:07:10.480ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
64122025-02-28T12:07:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
64132025-02-28T12:07:10.481ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64142025-02-28T12:07:10.481ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1265, tv_nsec: 117897068 } })
64152025-02-28T12:07:10.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64162025-02-28T12:07:10.482ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
64172025-02-28T12:07:10.482ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64182025-02-28T12:07:10.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64192025-02-28T12:07:10.486ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64202025-02-28T12:07:10.490ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
64212025-02-28T12:07:10.490ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
64222025-02-28T12:07:10.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
64232025-02-28T12:07:10.493ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64242025-02-28T12:07:10.495ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
64252025-02-28T12:07:10.495ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64262025-02-28T12:07:10.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64272025-02-28T12:07:10.518ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
64282025-02-28T12:07:10.518ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
64292025-02-28T12:07:10.518ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
64302025-02-28T12:07:10.518ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
64312025-02-28T12:07:10.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
64322025-02-28T12:07:10.519ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
64332025-02-28T12:07:10.519ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
64342025-02-28T12:07:10.519ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
64352025-02-28T12:07:10.519ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
64362025-02-28T12:07:10.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
64372025-02-28T12:07:10.522ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
64382025-02-28T12:07:10.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
64392025-02-28T12:07:10.524ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 66.041254ms iteration = 18 status = {"physical_disks_added":0}
64402025-02-28T12:07:10.524ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64412025-02-28T12:07:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
64422025-02-28T12:07:10.528ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
64432025-02-28T12:07:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
64442025-02-28T12:07:10.530ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64452025-02-28T12:07:10.530ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
64462025-02-28T12:07:10.530ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 0 version = 2
64472025-02-28T12:07:10.531ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
64482025-02-28T12:07:10.531ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 71.404073ms iteration = 17 status = {"errors":[],"finish_invoked_ok":[]}
64492025-02-28T12:07:10.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
64502025-02-28T12:07:10.531ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 71.965968ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
64512025-02-28T12:07:10.532ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 72.786217ms iteration = 17 status = {"errors":[],"garbage_collect_requested":[]}
64522025-02-28T12:07:10.532ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 73.057894ms iteration = 17 status = {"errors":[],"requests_created_ok":[]}
64532025-02-28T12:07:10.533ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
64542025-02-28T12:07:10.533ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
64552025-02-28T12:07:10.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
64562025-02-28T12:07:10.533ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
64572025-02-28T12:07:10.533ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
64582025-02-28T12:07:10.533ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 74.16643ms iteration = 9 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
64592025-02-28T12:07:10.534ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64602025-02-28T12:07:10.535ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64612025-02-28T12:07:10.536ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
64622025-02-28T12:07:10.536ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
64632025-02-28T12:07:10.536ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
64642025-02-28T12:07:10.536ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
64652025-02-28T12:07:10.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
64662025-02-28T12:07:10.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
64672025-02-28T12:07:10.537ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
64682025-02-28T12:07:10.538ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
64692025-02-28T12:07:10.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
64702025-02-28T12:07:10.539ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
64712025-02-28T12:07:10.539ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
64722025-02-28T12:07:10.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
64732025-02-28T12:07:10.540ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 81.59123ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
64742025-02-28T12:07:10.540ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64752025-02-28T12:07:10.540ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
64762025-02-28T12:07:10.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64772025-02-28T12:07:10.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
64782025-02-28T12:07:10.541ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
64792025-02-28T12:07:10.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
64802025-02-28T12:07:10.542ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
64812025-02-28T12:07:10.542ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
64822025-02-28T12:07:10.542ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 83.575228ms iteration = 9 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
64832025-02-28T12:07:10.542ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
64842025-02-28T12:07:10.542ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 83.70946ms iteration = 17 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
64852025-02-28T12:07:10.542ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64862025-02-28T12:07:10.543ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64872025-02-28T12:07:10.543ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 1 version = 2
64882025-02-28T12:07:10.543ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
64892025-02-28T12:07:10.543ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-02-28 11:59:52.542241 UTC dns_group = external generation = 2
64902025-02-28T12:07:10.543ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 88.457026ms iteration = 9 status = {"generation":2}
64912025-02-28T12:07:10.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64922025-02-28T12:07:10.545ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
64932025-02-28T12:07:10.545ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64942025-02-28T12:07:10.545ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64952025-02-28T12:07:10.547ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64962025-02-28T12:07:10.549ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
64972025-02-28T12:07:10.549ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64982025-02-28T12:07:10.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64992025-02-28T12:07:10.551ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 92.80968ms iteration = 17 status = 0
65002025-02-28T12:07:10.554ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
65012025-02-28T12:07:10.554ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 97.287188ms iteration = 49 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
65022025-02-28T12:07:10.559ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
65032025-02-28T12:07:10.559ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
65042025-02-28T12:07:10.575ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65052025-02-28T12:07:10.576ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
65062025-02-28T12:07:10.576ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
65072025-02-28T12:07:10.577ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
65082025-02-28T12:07:10.578ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
65092025-02-28T12:07:10.578ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
65102025-02-28T12:07:10.578ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
65112025-02-28T12:07:10.578ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
65122025-02-28T12:07:10.583ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
65132025-02-28T12:07:10.583ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
65142025-02-28T12:07:10.583ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 124.477605ms iteration = 17 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
65152025-02-28T12:07:10.589ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
65162025-02-28T12:07:10.589ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
65172025-02-28T12:07:10.591ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
65182025-02-28T12:07:10.591ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
65192025-02-28T12:07:10.604ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-28T11:59:08.719399Z, time_modified: 2025-02-28T11:59:08.719399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }) }
65202025-02-28T12:07:10.604ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 149.358066ms iteration = 9 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]}},"default_endpoint":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
65212025-02-28T12:07:10.662ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
65222025-02-28T12:07:10.662ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
65232025-02-28T12:07:10.662ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b03dd217-fea3-4974-ad09-57896eb220c1", "content-length": "26", "date": "Fri, 28 Feb 2025 12:07:10 GMT"} })
65242025-02-28T12:07:10.662ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
65252025-02-28T12:07:10.664ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
65262025-02-28T12:07:10.701ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
65272025-02-28T12:07:10.701ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
65282025-02-28T12:07:10.701ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7c95d26-cad4-4700-b983-e0fa277eca82", "content-length": "26", "date": "Fri, 28 Feb 2025 12:07:10 GMT"} })
65292025-02-28T12:07:10.701ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
65302025-02-28T12:07:10.703ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
65312025-02-28T12:07:10.708ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
65322025-02-28T12:07:10.708ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 250.165579ms iteration = 9 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
65332025-02-28T12:07:10.839ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
65342025-02-28T12:07:10.839ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
65352025-02-28T12:07:10.839ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c1f56db-2fb5-40e6-819d-ac2ab587c2f7", "content-length": "372", "date": "Fri, 28 Feb 2025 12:07:10 GMT"} })
65362025-02-28T12:07:10.839ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65372025-02-28T12:07:10.840ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65382025-02-28T12:07:10.841ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
65392025-02-28T12:07:10.841ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 382.626097ms iteration = 17 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
65402025-02-28T12:07:10.841ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65412025-02-28T12:07:10.842ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65422025-02-28T12:07:10.845ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
65432025-02-28T12:07:10.845ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
65442025-02-28T12:07:10.845ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
65452025-02-28T12:07:10.845ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
65462025-02-28T12:07:10.845ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
65472025-02-28T12:07:10.846ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
65482025-02-28T12:07:10.847ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65492025-02-28T12:07:10.849ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
65502025-02-28T12:07:10.849ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
65512025-02-28T12:07:10.849ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-02-28 11:59:51.915187 UTC dns_group = internal generation = 1
65522025-02-28T12:07:10.849ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 394.612926ms iteration = 9 status = {"generation":1}
65532025-02-28T12:07:10.853ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
65542025-02-28T12:07:10.853ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
65552025-02-28T12:07:10.853ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
65562025-02-28T12:07:10.882ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
65572025-02-28T12:07:10.883ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0e804f0-33ac-41e8-a0a2-15240a7bab8f", "content-length": "1", "date": "Fri, 28 Feb 2025 12:07:10 GMT"} })
65582025-02-28T12:07:10.884ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
65592025-02-28T12:07:10.885ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 425.348248ms iteration = 17 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
65602025-02-28T12:07:10.885ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
65612025-02-28T12:07:10.887ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 432.614403ms iteration = 17 status = null
65622025-02-28T12:07:10.892ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
65632025-02-28T12:07:10.914ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
65642025-02-28T12:07:10.914ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
65652025-02-28T12:07:10.914ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "a5c8905a-5e29-458e-885d-ebca5322abc5", "date": "Fri, 28 Feb 2025 12:07:10 GMT"} })
65662025-02-28T12:07:10.914ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65672025-02-28T12:07:10.914ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65682025-02-28T12:07:10.917ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65692025-02-28T12:07:10.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65702025-02-28T12:07:10.920ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65712025-02-28T12:07:10.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65722025-02-28T12:07:10.924ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
65732025-02-28T12:07:10.924ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
65742025-02-28T12:07:10.924ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "0b721d0e-f1ad-4f7b-9388-e87b25648ce1", "date": "Fri, 28 Feb 2025 12:07:10 GMT"} })
65752025-02-28T12:07:10.926ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 467.821114ms iteration = 17 status = {}
65762025-02-28T12:07:10.934ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
65772025-02-28T12:07:10.934ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
65782025-02-28T12:07:10.935ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "63ed6b31-2468-49a7-b26f-1d9813e57003", "content-length": "26", "date": "Fri, 28 Feb 2025 12:07:10 GMT"} })
65792025-02-28T12:07:10.935ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
65802025-02-28T12:07:10.937ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
65812025-02-28T12:07:10.970ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
65822025-02-28T12:07:11.099ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 644.370776ms iteration = 17 status = {}
65832025-02-28T12:07:11.129ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
65842025-02-28T12:07:11.129ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f162ed1-7728-4c3e-a726-eda93d7c7a88", "content-length": "691", "date": "Fri, 28 Feb 2025 12:07:10 GMT"} })
65852025-02-28T12:07:11.129ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
65862025-02-28T12:07:11.129ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
65872025-02-28T12:07:11.129ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
65882025-02-28T12:07:11.133ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 674.494753ms iteration = 17 status = {}
65892025-02-28T12:07:11.212ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
65902025-02-28T12:07:11.212ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "3a72a6c5-8899-40b6-bc3b-176a96552b01", "date": "Fri, 28 Feb 2025 12:07:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
65912025-02-28T12:07:11.212ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
65922025-02-28T12:07:11.214ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "bc516c21-fec4-4361-9903-42ba316929aa", "date": "Fri, 28 Feb 2025 12:07:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
65932025-02-28T12:07:11.214ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
65942025-02-28T12:07:11.216ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
65952025-02-28T12:07:11.216ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 761.950884ms iteration = 11 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
65962025-02-28T12:07:11.245ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
65972025-02-28T12:07:11.373ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
65982025-02-28T12:07:11.374ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "f03dd70b-526b-43f1-9136-10120da5bf9d", "date": "Fri, 28 Feb 2025 12:07:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
65992025-02-28T12:07:11.374ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
66002025-02-28T12:07:11.376ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "963dd2ca-260f-4eb7-9b19-d485051941be", "date": "Fri, 28 Feb 2025 12:07:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
66012025-02-28T12:07:11.376ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
66022025-02-28T12:07:11.376ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66032025-02-28T12:07:11.376ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66042025-02-28T12:07:11.378ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "3995df47-0c64-4a42-bc29-252dfec6bb89", "date": "Fri, 28 Feb 2025 12:07:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
66052025-02-28T12:07:11.378ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
66062025-02-28T12:07:11.378ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
66072025-02-28T12:07:11.378ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66082025-02-28T12:07:11.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66092025-02-28T12:07:11.380ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
66102025-02-28T12:07:11.380ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 925.271156ms iteration = 11 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
66112025-02-28T12:07:11.388ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
66122025-02-28T12:07:11.388ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7143680-ba3d-439f-9ba2-1b98dbd69e0f", "content-length": "134", "date": "Fri, 28 Feb 2025 12:07:11 GMT"} })
66132025-02-28T12:07:11.388ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
66142025-02-28T12:07:11.388ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
66152025-02-28T12:07:11.389ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
66162025-02-28T12:07:11.389ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
66172025-02-28T12:07:11.389ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66182025-02-28T12:07:11.389ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66192025-02-28T12:07:11.389ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66202025-02-28T12:07:11.389ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
66212025-02-28T12:07:11.389ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66222025-02-28T12:07:11.389ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
66232025-02-28T12:07:11.389ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
66242025-02-28T12:07:11.389ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66252025-02-28T12:07:11.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66262025-02-28T12:07:11.392ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
66272025-02-28T12:07:11.392ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "5950f338-cabe-407d-aa41-438c73426816", "content-length": "2", "date": "Fri, 28 Feb 2025 12:07:11 GMT"} })
66282025-02-28T12:07:11.392ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
66292025-02-28T12:07:11.392ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "58dc42a8-3b1e-4467-8997-72d090fa7dfe", "content-length": "51", "date": "Fri, 28 Feb 2025 12:07:11 GMT"} })
66302025-02-28T12:07:11.392ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
66312025-02-28T12:07:11.392ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
66322025-02-28T12:07:11.483ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "f8f27fa1-4aec-40c5-8250-bedc02d91f60", "date": "Fri, 28 Feb 2025 12:07:11 GMT"} })
66332025-02-28T12:07:11.483ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
66342025-02-28T12:07:11.483ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
66352025-02-28T12:07:11.483ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "925f79bb-6a2d-4a13-a5b9-d293aa393bca", "content-length": "682", "date": "Fri, 28 Feb 2025 12:07:10 GMT"} })
66362025-02-28T12:07:11.483ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66372025-02-28T12:07:11.484ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66382025-02-28T12:07:11.487ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66392025-02-28T12:07:11.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66402025-02-28T12:07:11.491ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66412025-02-28T12:07:11.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66422025-02-28T12:07:11.493ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66432025-02-28T12:07:11.493ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66442025-02-28T12:07:11.500ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 1.041749738s iteration = 17 status = {}
66452025-02-28T12:07:14.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 281 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = cac36b02-29fd-4793-b600-e091c1b5af97 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
66462025-02-28T12:07:20.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
66472025-02-28T12:07:20.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66482025-02-28T12:07:20.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66492025-02-28T12:07:20.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 50 reason = Timeout
66502025-02-28T12:07:20.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66512025-02-28T12:07:20.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66522025-02-28T12:07:20.461ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66532025-02-28T12:07:20.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66542025-02-28T12:07:20.493ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.068393ms iteration = 50 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
66552025-02-28T12:07:20.494ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66562025-02-28T12:07:20.496ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
66572025-02-28T12:07:20.496ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66582025-02-28T12:07:20.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66592025-02-28T12:07:20.498ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66602025-02-28T12:07:20.499ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66612025-02-28T12:07:20.500ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66622025-02-28T12:07:20.503ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
66632025-02-28T12:07:20.503ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66642025-02-28T12:07:20.503ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66652025-02-28T12:07:20.516ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
66662025-02-28T12:07:20.516ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 58.573644ms iteration = 50 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
66672025-02-28T12:07:24.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 260 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = b911df5a-6050-4ef1-b04a-cffb87ff3be5 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
66682025-02-28T12:07:30.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
66692025-02-28T12:07:30.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66702025-02-28T12:07:30.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66712025-02-28T12:07:30.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66722025-02-28T12:07:30.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 51 reason = Timeout
66732025-02-28T12:07:30.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66742025-02-28T12:07:30.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66752025-02-28T12:07:30.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66762025-02-28T12:07:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66772025-02-28T12:07:30.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
66782025-02-28T12:07:30.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66792025-02-28T12:07:30.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66802025-02-28T12:07:30.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66812025-02-28T12:07:30.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66822025-02-28T12:07:30.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66832025-02-28T12:07:30.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
66842025-02-28T12:07:30.468ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66852025-02-28T12:07:30.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66862025-02-28T12:07:30.480ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
66872025-02-28T12:07:30.480ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.864536ms iteration = 51 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
66882025-02-28T12:07:30.491ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.389502ms iteration = 51 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
66892025-02-28T12:07:34.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 294 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 63dbee2e-4d3d-49ee-84e6-ad3bd4efe244 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
66902025-02-28T12:07:39.254ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = b3ecb6f7-ea6e-450f-a716-804cf74f80c8 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
66912025-02-28T12:07:39.254ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = b3ecb6f7-ea6e-450f-a716-804cf74f80c8 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
66922025-02-28T12:07:39.256ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2335 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = b3ecb6f7-ea6e-450f-a716-804cf74f80c8 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
66932025-02-28T12:07:39.257ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 6e5dec42-0a4b-46b1-b6b0-20972902a5b8 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
66942025-02-28T12:07:39.257ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 6e5dec42-0a4b-46b1-b6b0-20972902a5b8 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
66952025-02-28T12:07:39.259ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2151 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 6e5dec42-0a4b-46b1-b6b0-20972902a5b8 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
66962025-02-28T12:07:40.454ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
66972025-02-28T12:07:40.454ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
66982025-02-28T12:07:40.454ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
66992025-02-28T12:07:40.454ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
67002025-02-28T12:07:40.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
67012025-02-28T12:07:40.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
67022025-02-28T12:07:40.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 52 reason = Timeout
67032025-02-28T12:07:40.457ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
67042025-02-28T12:07:40.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67052025-02-28T12:07:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67062025-02-28T12:07:40.457ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1295, tv_nsec: 93680464 } })
67072025-02-28T12:07:40.457ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
67082025-02-28T12:07:40.458ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1295, tv_nsec: 94561545 } })
67092025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 18 reason = Timeout
67102025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 52 reason = Timeout
67112025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 18 reason = Timeout
67122025-02-28T12:07:40.458ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
67132025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67142025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
67152025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
67162025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 18 reason = Timeout
67172025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
67182025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 18 reason = Timeout
67192025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 18 reason = Timeout
67202025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
67212025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
67222025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
67232025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
67242025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
67252025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 19 reason = Timeout
67262025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
67272025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 18 reason = Timeout
67282025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
67292025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
67302025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
67312025-02-28T12:07:40.458ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
67322025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67332025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
67342025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
67352025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
67362025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
67372025-02-28T12:07:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
67382025-02-28T12:07:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
67392025-02-28T12:07:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
67402025-02-28T12:07:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
67412025-02-28T12:07:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
67422025-02-28T12:07:40.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67432025-02-28T12:07:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 18 reason = Timeout
67442025-02-28T12:07:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
67452025-02-28T12:07:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 18 reason = Timeout
67462025-02-28T12:07:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
67472025-02-28T12:07:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 18 reason = Timeout
67482025-02-28T12:07:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
67492025-02-28T12:07:40.459ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 18 reason = Timeout
67502025-02-28T12:07:40.459ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
67512025-02-28T12:07:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 18 reason = Timeout
67522025-02-28T12:07:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
67532025-02-28T12:07:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
67542025-02-28T12:07:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
67552025-02-28T12:07:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
67562025-02-28T12:07:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
67572025-02-28T12:07:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
67582025-02-28T12:07:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67592025-02-28T12:07:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67602025-02-28T12:07:40.460ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67612025-02-28T12:07:40.461ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.14208ms iteration = 18 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
67622025-02-28T12:07:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
67632025-02-28T12:07:40.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67642025-02-28T12:07:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67652025-02-28T12:07:40.462ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
67662025-02-28T12:07:40.462ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
67672025-02-28T12:07:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
67682025-02-28T12:07:40.464ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67692025-02-28T12:07:40.465ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67702025-02-28T12:07:40.466ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67712025-02-28T12:07:40.466ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
67722025-02-28T12:07:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
67732025-02-28T12:07:40.467ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
67742025-02-28T12:07:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
67752025-02-28T12:07:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
67762025-02-28T12:07:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
67772025-02-28T12:07:40.468ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
67782025-02-28T12:07:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
67792025-02-28T12:07:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
67802025-02-28T12:07:40.468ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
67812025-02-28T12:07:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
67822025-02-28T12:07:40.470ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 12.45032ms iteration = 52 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
67832025-02-28T12:07:40.471ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67842025-02-28T12:07:40.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67852025-02-28T12:07:40.473ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
67862025-02-28T12:07:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
67872025-02-28T12:07:40.473ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
67882025-02-28T12:07:40.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
67892025-02-28T12:07:40.474ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
67902025-02-28T12:07:40.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
67912025-02-28T12:07:40.474ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67922025-02-28T12:07:40.476ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
67932025-02-28T12:07:40.476ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67942025-02-28T12:07:40.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67952025-02-28T12:07:40.493ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
67962025-02-28T12:07:40.493ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 36.225674ms iteration = 52 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
67972025-02-28T12:07:40.501ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
67982025-02-28T12:07:40.501ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
67992025-02-28T12:07:40.502ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
68002025-02-28T12:07:40.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
68012025-02-28T12:07:40.504ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68022025-02-28T12:07:40.507ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68032025-02-28T12:07:40.507ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
68042025-02-28T12:07:40.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68052025-02-28T12:07:40.507ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
68062025-02-28T12:07:40.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
68072025-02-28T12:07:40.510ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68082025-02-28T12:07:40.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68092025-02-28T12:07:40.514ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 54.550997ms iteration = 18 status = {"errors":[],"garbage_collect_requested":[]}
68102025-02-28T12:07:40.514ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 54.567507ms iteration = 18 status = {"errors":[],"finish_invoked_ok":[]}
68112025-02-28T12:07:40.514ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68122025-02-28T12:07:40.515ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 56.684799ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
68132025-02-28T12:07:40.515ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1295, tv_nsec: 151592572 } })
68142025-02-28T12:07:40.515ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
68152025-02-28T12:07:40.515ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
68162025-02-28T12:07:40.516ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68172025-02-28T12:07:40.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
68182025-02-28T12:07:40.518ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
68192025-02-28T12:07:40.518ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68202025-02-28T12:07:40.518ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 59.040256ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
68212025-02-28T12:07:40.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68222025-02-28T12:07:40.519ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68232025-02-28T12:07:40.519ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
68242025-02-28T12:07:40.519ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68252025-02-28T12:07:40.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68262025-02-28T12:07:40.521ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
68272025-02-28T12:07:40.521ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
68282025-02-28T12:07:40.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68292025-02-28T12:07:40.522ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 63.432992ms iteration = 18 status = 0
68302025-02-28T12:07:40.522ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 62.808676ms iteration = 18 status = {"errors":[],"requests_created_ok":[]}
68312025-02-28T12:07:40.523ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68322025-02-28T12:07:40.525ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
68332025-02-28T12:07:40.525ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
68342025-02-28T12:07:40.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68352025-02-28T12:07:40.533ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
68362025-02-28T12:07:40.533ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 74.905688ms iteration = 18 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
68372025-02-28T12:07:40.547ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
68382025-02-28T12:07:40.547ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
68392025-02-28T12:07:40.547ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
68402025-02-28T12:07:40.547ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
68412025-02-28T12:07:40.559ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
68422025-02-28T12:07:40.559ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
68432025-02-28T12:07:40.562ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
68442025-02-28T12:07:40.562ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68452025-02-28T12:07:40.565ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68462025-02-28T12:07:40.567ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
68472025-02-28T12:07:40.567ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
68482025-02-28T12:07:40.567ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68492025-02-28T12:07:40.584ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
68502025-02-28T12:07:40.584ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
68512025-02-28T12:07:40.584ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 126.384491ms iteration = 18 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
68522025-02-28T12:07:40.597ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
68532025-02-28T12:07:40.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
68542025-02-28T12:07:40.601ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 141.551897ms iteration = 18 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
68552025-02-28T12:07:40.606ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
68562025-02-28T12:07:40.606ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 147.750216ms iteration = 18 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
68572025-02-28T12:07:40.635ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
68582025-02-28T12:07:40.635ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
68592025-02-28T12:07:40.635ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "612ec915-b47a-405e-87b3-bf37deca4d52", "content-length": "26", "date": "Fri, 28 Feb 2025 12:07:40 GMT"} })
68602025-02-28T12:07:40.635ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
68612025-02-28T12:07:40.637ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
68622025-02-28T12:07:40.639ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
68632025-02-28T12:07:40.639ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
68642025-02-28T12:07:40.640ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdc52bc7-0b15-49ca-96a5-9fe95658c77c", "content-length": "26", "date": "Fri, 28 Feb 2025 12:07:40 GMT"} })
68652025-02-28T12:07:40.640ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
68662025-02-28T12:07:40.641ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
68672025-02-28T12:07:40.649ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
68682025-02-28T12:07:40.649ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
68692025-02-28T12:07:40.650ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2993881-9f77-4960-8d56-eec26cb500d9", "content-length": "26", "date": "Fri, 28 Feb 2025 12:07:40 GMT"} })
68702025-02-28T12:07:40.650ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
68712025-02-28T12:07:40.651ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 193.122531ms iteration = 19 status = {"physical_disks_added":0}
68722025-02-28T12:07:40.653ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
68732025-02-28T12:07:40.714ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
68742025-02-28T12:07:40.714ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
68752025-02-28T12:07:40.715ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "56892a90-7dae-4951-ab99-c4c4281f4ce9", "content-length": "372", "date": "Fri, 28 Feb 2025 12:07:40 GMT"} })
68762025-02-28T12:07:40.715ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68772025-02-28T12:07:40.715ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68782025-02-28T12:07:40.736ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
68792025-02-28T12:07:40.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
68802025-02-28T12:07:40.739ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68812025-02-28T12:07:40.741ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
68822025-02-28T12:07:40.741ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
68832025-02-28T12:07:40.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
68842025-02-28T12:07:40.743ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68852025-02-28T12:07:40.746ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
68862025-02-28T12:07:40.746ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
68872025-02-28T12:07:40.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
68882025-02-28T12:07:40.763ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
68892025-02-28T12:07:40.764ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "77bd708a-ff7d-41c9-a1a1-63b81e7e3a51", "content-length": "1", "date": "Fri, 28 Feb 2025 12:07:40 GMT"} })
68902025-02-28T12:07:40.765ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
68912025-02-28T12:07:40.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
68922025-02-28T12:07:40.771ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 317.120776ms iteration = 18 status = null
68932025-02-28T12:07:40.794ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
68942025-02-28T12:07:40.794ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
68952025-02-28T12:07:40.794ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "2326dc7f-3c2e-4ded-b9d6-8e601fe9b2a3", "date": "Fri, 28 Feb 2025 12:07:40 GMT"} })
68962025-02-28T12:07:40.794ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68972025-02-28T12:07:40.795ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68982025-02-28T12:07:40.798ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68992025-02-28T12:07:40.798ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69002025-02-28T12:07:40.801ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69012025-02-28T12:07:40.802ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69022025-02-28T12:07:40.808ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
69032025-02-28T12:07:40.809ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
69042025-02-28T12:07:40.809ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "94a8f932-f0cd-465b-a695-e0867783d3c9", "date": "Fri, 28 Feb 2025 12:07:40 GMT"} })
69052025-02-28T12:07:40.811ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 353.239227ms iteration = 18 status = {}
69062025-02-28T12:07:40.833ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 378.799294ms iteration = 18 status = {}
69072025-02-28T12:07:40.898ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
69082025-02-28T12:07:40.898ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "45f648e8-4161-4d7a-905a-211719ac0dd5", "content-length": "691", "date": "Fri, 28 Feb 2025 12:07:40 GMT"} })
69092025-02-28T12:07:40.898ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
69102025-02-28T12:07:40.898ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
69112025-02-28T12:07:40.898ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
69122025-02-28T12:07:40.900ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 441.776364ms iteration = 18 status = {}
69132025-02-28T12:07:40.924ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69142025-02-28T12:07:40.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69152025-02-28T12:07:40.926ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
69162025-02-28T12:07:40.926ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69172025-02-28T12:07:40.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69182025-02-28T12:07:40.936ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
69192025-02-28T12:07:40.936ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "232879ab-d94f-4905-8df6-6ae54c7cdbff", "content-length": "134", "date": "Fri, 28 Feb 2025 12:07:40 GMT"} })
69202025-02-28T12:07:40.936ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
69212025-02-28T12:07:40.936ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
69222025-02-28T12:07:40.937ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
69232025-02-28T12:07:40.937ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
69242025-02-28T12:07:40.937ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
69252025-02-28T12:07:40.937ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
69262025-02-28T12:07:40.937ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
69272025-02-28T12:07:40.937ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
69282025-02-28T12:07:40.937ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
69292025-02-28T12:07:40.937ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
69302025-02-28T12:07:40.937ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
69312025-02-28T12:07:40.937ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69322025-02-28T12:07:40.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69332025-02-28T12:07:40.940ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
69342025-02-28T12:07:40.940ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f061537-1965-4842-b562-560a39af4d29", "content-length": "2", "date": "Fri, 28 Feb 2025 12:07:40 GMT"} })
69352025-02-28T12:07:40.940ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
69362025-02-28T12:07:40.940ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bb6d8c0-e2fd-4f86-9bcb-1bd6c0fe923b", "content-length": "51", "date": "Fri, 28 Feb 2025 12:07:40 GMT"} })
69372025-02-28T12:07:40.940ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
69382025-02-28T12:07:40.940ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
69392025-02-28T12:07:41.047ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "f635b8dd-1b6e-40c2-89a8-044fa68d5f18", "date": "Fri, 28 Feb 2025 12:07:40 GMT"} })
69402025-02-28T12:07:41.047ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
69412025-02-28T12:07:41.047ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
69422025-02-28T12:07:41.047ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "4339b33a-e8cc-43f0-9440-3370e5d2dda0", "content-length": "682", "date": "Fri, 28 Feb 2025 12:07:40 GMT"} })
69432025-02-28T12:07:41.048ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69442025-02-28T12:07:41.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69452025-02-28T12:07:41.051ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69462025-02-28T12:07:41.052ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69472025-02-28T12:07:41.056ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69482025-02-28T12:07:41.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69492025-02-28T12:07:41.058ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69502025-02-28T12:07:41.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69512025-02-28T12:07:41.067ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 608.703734ms iteration = 18 status = {}
69522025-02-28T12:07:41.272ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
69532025-02-28T12:07:41.272ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::a]:12221
69542025-02-28T12:07:41.302ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
69552025-02-28T12:07:41.303ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::a]:56444
69562025-02-28T12:07:41.303ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:56444 req_id = f8670144-24fe-44e5-a9bc-9ef1f8b6bd2b roles = RoleSet { roles: {} } uri = /metrics/producers
69572025-02-28T12:07:41.303ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:56444 req_id = f8670144-24fe-44e5-a9bc-9ef1f8b6bd2b resource = Database result = Ok(()) uri = /metrics/producers
69582025-02-28T12:07:41.342ZINFOnexus: assigned collector to new producer collector_id = a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79 file = nexus/src/app/oximeter.rs:76 producer_id = c1b34bc5-1c2f-4d3c-808c-b7a441aac295
69592025-02-28T12:07:41.342ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 39371 local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:56444 req_id = f8670144-24fe-44e5-a9bc-9ef1f8b6bd2b response_code = 201 uri = /metrics/producers
69602025-02-28T12:07:41.342ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "f8670144-24fe-44e5-a9bc-9ef1f8b6bd2b", "content-length": "41", "date": "Fri, 28 Feb 2025 12:07:41 GMT"} })
69612025-02-28T12:07:41.343ZDEBGnexus (producer-registration-task): registered with nexus successfully
69622025-02-28T12:07:41.343ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
69632025-02-28T12:07:44.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 301 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 08d51855-211c-4b97-9510-0f1af8274365 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
69642025-02-28T12:07:50.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 53 reason = Timeout
69652025-02-28T12:07:50.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69662025-02-28T12:07:50.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69672025-02-28T12:07:50.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69682025-02-28T12:07:50.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 53 reason = Timeout
69692025-02-28T12:07:50.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
69702025-02-28T12:07:50.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
69712025-02-28T12:07:50.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
69722025-02-28T12:07:50.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69732025-02-28T12:07:50.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69742025-02-28T12:07:50.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69752025-02-28T12:07:50.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69762025-02-28T12:07:50.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69772025-02-28T12:07:50.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
69782025-02-28T12:07:50.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69792025-02-28T12:07:50.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69802025-02-28T12:07:50.483ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
69812025-02-28T12:07:50.483ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.989061ms iteration = 53 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
69822025-02-28T12:07:50.494ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
69832025-02-28T12:07:50.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
69842025-02-28T12:07:50.524ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 66.045615ms iteration = 53 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
69852025-02-28T12:07:54.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 251 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 12ba5740-c4bc-47ff-8452-f834811ed9fe response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
69862025-02-28T12:08:00.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 54 reason = Timeout
69872025-02-28T12:08:00.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69882025-02-28T12:08:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69892025-02-28T12:08:00.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 54 reason = Timeout
69902025-02-28T12:08:00.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
69912025-02-28T12:08:00.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69922025-02-28T12:08:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
69932025-02-28T12:08:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
69942025-02-28T12:08:00.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69952025-02-28T12:08:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69962025-02-28T12:08:00.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69972025-02-28T12:08:00.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69982025-02-28T12:08:00.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69992025-02-28T12:08:00.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
70002025-02-28T12:08:00.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70012025-02-28T12:08:00.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70022025-02-28T12:08:00.481ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
70032025-02-28T12:08:00.481ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.173213ms iteration = 54 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
70042025-02-28T12:08:00.495ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
70052025-02-28T12:08:00.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
70062025-02-28T12:08:00.497ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 38.141581ms iteration = 54 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
70072025-02-28T12:08:04.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 265 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 17f24226-8ade-4e7b-8ff8-1952331ac0da response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
70082025-02-28T12:08:09.253ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = ad32580b-b0df-4c23-9c05-5c5004f579cb roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
70092025-02-28T12:08:09.254ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = ad32580b-b0df-4c23-9c05-5c5004f579cb resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
70102025-02-28T12:08:09.256ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2749 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = ad32580b-b0df-4c23-9c05-5c5004f579cb response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
70112025-02-28T12:08:09.256ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 62b9f8e2-d3c5-4b1e-8caa-1905a675322f roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
70122025-02-28T12:08:09.257ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 62b9f8e2-d3c5-4b1e-8caa-1905a675322f resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
70132025-02-28T12:08:09.258ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2015 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 62b9f8e2-d3c5-4b1e-8caa-1905a675322f response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
70142025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 10 reason = Timeout
70152025-02-28T12:08:10.454ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-28 11:58:10.454150447 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
70162025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 10 reason = Timeout
70172025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
70182025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
70192025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
70202025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 12 reason = Timeout
70212025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 10 reason = Timeout
70222025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 10 reason = Timeout
70232025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 19 reason = Timeout
70242025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
70252025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 10 reason = Timeout
70262025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
70272025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
70282025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 10 reason = Timeout
70292025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
70302025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 19 reason = Timeout
70312025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
70322025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 12 reason = Timeout
70332025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
70342025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
70352025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
70362025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
70372025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
70382025-02-28T12:08:10.454ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5cefabf1-6831-46ff-98eb-1a3b5abad8b9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1325, tv_nsec: 91286306 } })
70392025-02-28T12:08:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
70402025-02-28T12:08:10.454ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5ebff2d9-48b6-4fe6-b852-684bb14b4955.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bddfb0f9-1d7b-45e4-b3a5-35aa6792956b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1325, tv_nsec: 91356978 } })
70412025-02-28T12:08:10.455ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
70422025-02-28T12:08:10.455ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.207659ms iteration = 10 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
70432025-02-28T12:08:10.455ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
70442025-02-28T12:08:10.455ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.338002ms iteration = 10 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
70452025-02-28T12:08:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
70462025-02-28T12:08:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
70472025-02-28T12:08:10.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 55 reason = Timeout
70482025-02-28T12:08:10.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 11 reason = Timeout
70492025-02-28T12:08:10.456ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
70502025-02-28T12:08:10.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70512025-02-28T12:08:10.456ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 19 reason = Timeout
70522025-02-28T12:08:10.456ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 9.97\u{b5}s iteration = 11 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
70532025-02-28T12:08:10.456ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 5 reason = Timeout
70542025-02-28T12:08:10.456ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
70552025-02-28T12:08:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
70562025-02-28T12:08:10.456ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
70572025-02-28T12:08:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70582025-02-28T12:08:10.456ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
70592025-02-28T12:08:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
70602025-02-28T12:08:10.456ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
70612025-02-28T12:08:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
70622025-02-28T12:08:10.457ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
70632025-02-28T12:08:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
70642025-02-28T12:08:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
70652025-02-28T12:08:10.457ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
70662025-02-28T12:08:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
70672025-02-28T12:08:10.457ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-28 11:58:10.454150447 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
70682025-02-28T12:08:10.457ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.364142ms iteration = 10 status = {"expiration":"2025-02-28T11:58:10.454150447Z","pruned":[]}
70692025-02-28T12:08:10.458ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1325, tv_nsec: 94386131 } })
70702025-02-28T12:08:10.458ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
70712025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 10 reason = Timeout
70722025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 19 reason = Timeout
70732025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
70742025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 19 reason = Timeout
70752025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
70762025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
70772025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 19 reason = Timeout
70782025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 55 reason = Timeout
70792025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70802025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
70812025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 19 reason = Timeout
70822025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
70832025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 20 reason = Timeout
70842025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 19 reason = Timeout
70852025-02-28T12:08:10.458ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
70862025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
70872025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
70882025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 19 reason = Timeout
70892025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
70902025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 19 reason = Timeout
70912025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
70922025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 10 reason = Timeout
70932025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
70942025-02-28T12:08:10.458ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
70952025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
70962025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 19 reason = Timeout
70972025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
70982025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
70992025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
71002025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
71012025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71022025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
71032025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71042025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
71052025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
71062025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
71072025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
71082025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
71092025-02-28T12:08:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
71102025-02-28T12:08:10.459ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1325, tv_nsec: 95346754 } })
71112025-02-28T12:08:10.459ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
71122025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 10 reason = Timeout
71132025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 19 reason = Timeout
71142025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
71152025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
71162025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 19 reason = Timeout
71172025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
71182025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 19 reason = Timeout
71192025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
71202025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 10 reason = Timeout
71212025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
71222025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 19 reason = Timeout
71232025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
71242025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 19 reason = Timeout
71252025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
71262025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
71272025-02-28T12:08:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
71282025-02-28T12:08:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
71292025-02-28T12:08:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
71302025-02-28T12:08:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
71312025-02-28T12:08:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
71322025-02-28T12:08:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
71332025-02-28T12:08:10.460ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71342025-02-28T12:08:10.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71352025-02-28T12:08:10.461ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.944439ms iteration = 19 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
71362025-02-28T12:08:10.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
71372025-02-28T12:08:10.462ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
71382025-02-28T12:08:10.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
71392025-02-28T12:08:10.462ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71402025-02-28T12:08:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
71412025-02-28T12:08:10.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71422025-02-28T12:08:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71432025-02-28T12:08:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
71442025-02-28T12:08:10.464ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
71452025-02-28T12:08:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
71462025-02-28T12:08:10.465ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71472025-02-28T12:08:10.467ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71482025-02-28T12:08:10.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
71492025-02-28T12:08:10.467ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
71502025-02-28T12:08:10.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
71512025-02-28T12:08:10.468ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71522025-02-28T12:08:10.468ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71532025-02-28T12:08:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71542025-02-28T12:08:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
71552025-02-28T12:08:10.470ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
71562025-02-28T12:08:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
71572025-02-28T12:08:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
71582025-02-28T12:08:10.471ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
71592025-02-28T12:08:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
71602025-02-28T12:08:10.473ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71612025-02-28T12:08:10.474ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71622025-02-28T12:08:10.475ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
71632025-02-28T12:08:10.475ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
71642025-02-28T12:08:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
71652025-02-28T12:08:10.476ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
71662025-02-28T12:08:10.476ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
71672025-02-28T12:08:10.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
71682025-02-28T12:08:10.480ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 20.590257ms iteration = 10 status = {"errors":[],"found_port_ok":[]}
71692025-02-28T12:08:10.483ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
71702025-02-28T12:08:10.484ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
71712025-02-28T12:08:10.487ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71722025-02-28T12:08:10.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71732025-02-28T12:08:10.503ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71742025-02-28T12:08:10.506ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
71752025-02-28T12:08:10.506ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
71762025-02-28T12:08:10.506ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
71772025-02-28T12:08:10.514ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71782025-02-28T12:08:10.515ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71792025-02-28T12:08:10.515ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71802025-02-28T12:08:10.517ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
71812025-02-28T12:08:10.517ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
71822025-02-28T12:08:10.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
71832025-02-28T12:08:10.517ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71842025-02-28T12:08:10.519ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
71852025-02-28T12:08:10.519ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
71862025-02-28T12:08:10.520ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
71872025-02-28T12:08:10.520ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
71882025-02-28T12:08:10.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
71892025-02-28T12:08:10.523ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
71902025-02-28T12:08:10.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
71912025-02-28T12:08:10.523ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71922025-02-28T12:08:10.523ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
71932025-02-28T12:08:10.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
71942025-02-28T12:08:10.525ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71952025-02-28T12:08:10.525ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
71962025-02-28T12:08:10.525ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
71972025-02-28T12:08:10.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
71982025-02-28T12:08:10.526ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71992025-02-28T12:08:10.527ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
72002025-02-28T12:08:10.527ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 69.250302ms iteration = 19 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
72012025-02-28T12:08:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
72022025-02-28T12:08:10.528ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72032025-02-28T12:08:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72042025-02-28T12:08:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
72052025-02-28T12:08:10.528ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
72062025-02-28T12:08:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
72072025-02-28T12:08:10.531ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
72082025-02-28T12:08:10.531ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
72092025-02-28T12:08:10.531ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
72102025-02-28T12:08:10.531ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72112025-02-28T12:08:10.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
72122025-02-28T12:08:10.532ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
72132025-02-28T12:08:10.532ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 72.978792ms iteration = 19 status = {"errors":[],"finish_invoked_ok":[]}
72142025-02-28T12:08:10.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
72152025-02-28T12:08:10.534ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
72162025-02-28T12:08:10.535ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
72172025-02-28T12:08:10.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
72182025-02-28T12:08:10.535ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72192025-02-28T12:08:10.537ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72202025-02-28T12:08:10.537ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
72212025-02-28T12:08:10.537ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
72222025-02-28T12:08:10.537ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
72232025-02-28T12:08:10.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
72242025-02-28T12:08:10.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
72252025-02-28T12:08:10.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
72262025-02-28T12:08:10.538ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
72272025-02-28T12:08:10.538ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
72282025-02-28T12:08:10.538ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
72292025-02-28T12:08:10.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
72302025-02-28T12:08:10.539ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 83.076315ms iteration = 5 status = {"nsuccess":5}
72312025-02-28T12:08:10.540ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 81.055916ms iteration = 19 status = {"errors":[],"garbage_collect_requested":[]}
72322025-02-28T12:08:10.541ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 81.336724ms iteration = 19 status = {"errors":[],"requests_created_ok":[]}
72332025-02-28T12:08:10.543ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72342025-02-28T12:08:10.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72352025-02-28T12:08:10.543ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
72362025-02-28T12:08:10.543ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
72372025-02-28T12:08:10.543ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 84.241643ms iteration = 10 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
72382025-02-28T12:08:10.549ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1325, tv_nsec: 185788987 } })
72392025-02-28T12:08:10.549ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
72402025-02-28T12:08:10.553ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
72412025-02-28T12:08:10.553ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 96.580662ms iteration = 55 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
72422025-02-28T12:08:10.555ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72432025-02-28T12:08:10.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
72442025-02-28T12:08:10.558ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
72452025-02-28T12:08:10.558ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
72462025-02-28T12:08:10.558ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
72472025-02-28T12:08:10.558ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
72482025-02-28T12:08:10.558ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
72492025-02-28T12:08:10.559ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 101.386758ms iteration = 55 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
72502025-02-28T12:08:10.562ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72512025-02-28T12:08:10.562ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72522025-02-28T12:08:10.565ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 106.893ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
72532025-02-28T12:08:10.565ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
72542025-02-28T12:08:10.565ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
72552025-02-28T12:08:10.565ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
72562025-02-28T12:08:10.565ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
72572025-02-28T12:08:10.565ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
72582025-02-28T12:08:10.565ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
72592025-02-28T12:08:10.565ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 107.133166ms iteration = 10 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
72602025-02-28T12:08:10.565ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
72612025-02-28T12:08:10.565ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
72622025-02-28T12:08:10.566ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 106.310207ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
72632025-02-28T12:08:10.566ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
72642025-02-28T12:08:10.569ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72652025-02-28T12:08:10.572ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
72662025-02-28T12:08:10.572ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
72672025-02-28T12:08:10.572ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
72682025-02-28T12:08:10.599ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
72692025-02-28T12:08:10.607ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
72702025-02-28T12:08:10.607ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
72712025-02-28T12:08:10.607ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a853c23-c6d4-4f7a-9556-c4c6e5272f6e", "content-length": "26", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} })
72722025-02-28T12:08:10.607ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
72732025-02-28T12:08:10.609ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
72742025-02-28T12:08:10.712ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
72752025-02-28T12:08:10.748ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
72762025-02-28T12:08:10.748ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
72772025-02-28T12:08:10.750ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6bac6f6-9db0-4ff3-8884-d11637eac86b", "content-length": "372", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} })
72782025-02-28T12:08:10.750ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
72792025-02-28T12:08:10.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
72802025-02-28T12:08:10.764ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72812025-02-28T12:08:10.767ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
72822025-02-28T12:08:10.767ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72832025-02-28T12:08:10.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
72842025-02-28T12:08:10.770ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72852025-02-28T12:08:10.772ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72862025-02-28T12:08:10.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72872025-02-28T12:08:10.774ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
72882025-02-28T12:08:10.774ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72892025-02-28T12:08:10.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
72902025-02-28T12:08:10.788ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
72912025-02-28T12:08:10.788ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
72922025-02-28T12:08:10.788ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "37da01c3-530a-4927-9ee1-6fd422598b3e", "content-length": "26", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} })
72932025-02-28T12:08:10.788ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
72942025-02-28T12:08:10.790ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
72952025-02-28T12:08:10.807ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
72962025-02-28T12:08:10.807ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
72972025-02-28T12:08:10.807ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9efcba72-cce3-46fc-81c4-c10afd05ea27", "content-length": "26", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} })
72982025-02-28T12:08:10.807ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
72992025-02-28T12:08:10.809ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
73002025-02-28T12:08:10.810ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
73012025-02-28T12:08:10.811ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
73022025-02-28T12:08:10.820ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
73032025-02-28T12:08:10.821ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "216c8d12-8aa2-43cf-8a98-b4e570dc9d1e", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
73042025-02-28T12:08:10.821ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
73052025-02-28T12:08:10.822ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
73062025-02-28T12:08:10.822ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73072025-02-28T12:08:10.823ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
73082025-02-28T12:08:10.823ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "9cd1bf7c-5793-4e5c-af24-463a42eb23a9", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
73092025-02-28T12:08:10.823ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
73102025-02-28T12:08:10.829ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73112025-02-28T12:08:10.831ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
73122025-02-28T12:08:10.831ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 376.434647ms iteration = 12 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
73132025-02-28T12:08:10.833ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
73142025-02-28T12:08:10.833ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
73152025-02-28T12:08:10.833ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
73162025-02-28T12:08:10.835ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
73172025-02-28T12:08:10.835ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
73182025-02-28T12:08:10.835ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
73192025-02-28T12:08:10.838ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73202025-02-28T12:08:10.841ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
73212025-02-28T12:08:10.841ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b7cda56-56d4-4e91-96db-9bb93fa170be", "content-length": "1", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} })
73222025-02-28T12:08:10.842ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
73232025-02-28T12:08:10.842ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
73242025-02-28T12:08:10.842ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
73252025-02-28T12:08:10.842ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
73262025-02-28T12:08:10.843ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
73272025-02-28T12:08:10.844ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
73282025-02-28T12:08:10.845ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
73292025-02-28T12:08:10.845ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 387.174537ms iteration = 19 status = 0
73302025-02-28T12:08:10.845ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73312025-02-28T12:08:10.847ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 393.052548ms iteration = 19 status = null
73322025-02-28T12:08:10.847ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73332025-02-28T12:08:10.847ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
73342025-02-28T12:08:10.847ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
73352025-02-28T12:08:10.848ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
73362025-02-28T12:08:10.850ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
73372025-02-28T12:08:10.850ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
73382025-02-28T12:08:10.851ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
73392025-02-28T12:08:10.854ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
73402025-02-28T12:08:10.854ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
73412025-02-28T12:08:10.854ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 395.620561ms iteration = 19 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
73422025-02-28T12:08:10.871ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 411.689358ms iteration = 19 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
73432025-02-28T12:08:10.873ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73442025-02-28T12:08:10.874ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
73452025-02-28T12:08:10.874ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
73462025-02-28T12:08:10.875ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "8d275cc1-a9fa-43c0-a947-86c784568bb4", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} })
73472025-02-28T12:08:10.875ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73482025-02-28T12:08:10.875ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73492025-02-28T12:08:10.876ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
73502025-02-28T12:08:10.876ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
73512025-02-28T12:08:10.876ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
73522025-02-28T12:08:10.878ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
73532025-02-28T12:08:10.878ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
73542025-02-28T12:08:10.878ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
73552025-02-28T12:08:10.878ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73562025-02-28T12:08:10.878ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
73572025-02-28T12:08:10.878ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73582025-02-28T12:08:10.880ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73592025-02-28T12:08:10.882ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73602025-02-28T12:08:10.883ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73612025-02-28T12:08:10.883ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
73622025-02-28T12:08:10.883ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
73632025-02-28T12:08:10.884ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
73642025-02-28T12:08:10.887ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
73652025-02-28T12:08:10.887ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
73662025-02-28T12:08:10.887ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
73672025-02-28T12:08:10.887ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
73682025-02-28T12:08:10.887ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-02-28 11:59:51.915187 UTC dns_group = internal generation = 1
73692025-02-28T12:08:10.887ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 433.005913ms iteration = 10 status = {"generation":1}
73702025-02-28T12:08:10.887ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "d5483cdd-555c-439d-b579-dffec7c09245", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} })
73712025-02-28T12:08:10.890ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 432.023009ms iteration = 19 status = {}
73722025-02-28T12:08:10.894ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-28T11:59:08.719399Z, time_modified: 2025-02-28T11:59:08.719399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }) }
73732025-02-28T12:08:10.894ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 439.907669ms iteration = 10 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]}},"default_endpoint":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
73742025-02-28T12:08:10.897ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
73752025-02-28T12:08:10.897ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 439.056949ms iteration = 10 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
73762025-02-28T12:08:10.897ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
73772025-02-28T12:08:10.897ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 0 version = 2
73782025-02-28T12:08:10.897ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
73792025-02-28T12:08:10.897ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
73802025-02-28T12:08:10.898ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
73812025-02-28T12:08:10.899ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73822025-02-28T12:08:10.903ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
73832025-02-28T12:08:10.903ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
73842025-02-28T12:08:10.903ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
73852025-02-28T12:08:10.905ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 1 version = 2
73862025-02-28T12:08:10.905ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
73872025-02-28T12:08:10.905ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-02-28 11:59:52.542241 UTC dns_group = external generation = 2
73882025-02-28T12:08:10.905ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 451.719154ms iteration = 10 status = {"generation":2}
73892025-02-28T12:08:10.916ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 462.340431ms iteration = 19 status = {}
73902025-02-28T12:08:10.917ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
73912025-02-28T12:08:10.917ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 459.484622ms iteration = 19 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
73922025-02-28T12:08:10.986ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 527.990516ms iteration = 20 status = {"physical_disks_added":0}
73932025-02-28T12:08:10.995ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
73942025-02-28T12:08:10.996ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "a972a4af-0edf-4086-a78d-9bf89a977a61", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
73952025-02-28T12:08:10.996ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
73962025-02-28T12:08:10.998ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "61d134ed-6f57-4acf-90bc-9b8b53068460", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
73972025-02-28T12:08:10.998ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
73982025-02-28T12:08:11.000ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "87bef488-55c6-44cc-bd89-87f818894aba", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
73992025-02-28T12:08:11.000ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
74002025-02-28T12:08:11.001ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
74012025-02-28T12:08:11.001ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 547.478066ms iteration = 12 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
74022025-02-28T12:08:11.003ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
74032025-02-28T12:08:11.004ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "92b36dd4-d2e3-4752-bc2f-7d2ad55d7ae2", "content-length": "691", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} })
74042025-02-28T12:08:11.004ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
74052025-02-28T12:08:11.004ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
74062025-02-28T12:08:11.004ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
74072025-02-28T12:08:11.006ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 548.320787ms iteration = 19 status = {}
74082025-02-28T12:08:11.037ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74092025-02-28T12:08:11.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74102025-02-28T12:08:11.040ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
74112025-02-28T12:08:11.040ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74122025-02-28T12:08:11.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74132025-02-28T12:08:11.050ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
74142025-02-28T12:08:11.050ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "2421f936-895e-4bea-b411-1dd64db9dd43", "content-length": "134", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} })
74152025-02-28T12:08:11.050ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
74162025-02-28T12:08:11.050ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
74172025-02-28T12:08:11.051ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
74182025-02-28T12:08:11.051ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
74192025-02-28T12:08:11.051ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74202025-02-28T12:08:11.051ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74212025-02-28T12:08:11.051ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74222025-02-28T12:08:11.051ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
74232025-02-28T12:08:11.051ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74242025-02-28T12:08:11.051ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
74252025-02-28T12:08:11.051ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
74262025-02-28T12:08:11.051ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74272025-02-28T12:08:11.052ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74282025-02-28T12:08:11.054ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
74292025-02-28T12:08:11.054ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffe7702f-5f1c-454d-9a8a-3a89c8581d5f", "content-length": "2", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} })
74302025-02-28T12:08:11.054ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
74312025-02-28T12:08:11.054ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "54fda962-2fd8-40eb-b0bd-4829717a83c1", "content-length": "51", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} })
74322025-02-28T12:08:11.054ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
74332025-02-28T12:08:11.054ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
74342025-02-28T12:08:11.161ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "32b671e3-f2f2-44fc-8a3d-c71bab98f3f3", "date": "Fri, 28 Feb 2025 12:08:10 GMT"} })
74352025-02-28T12:08:11.162ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
74362025-02-28T12:08:11.162ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
74372025-02-28T12:08:11.162ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "318a9fea-8019-48cb-8b6d-ee9576082349", "content-length": "682", "date": "Fri, 28 Feb 2025 12:08:11 GMT"} })
74382025-02-28T12:08:11.162ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74392025-02-28T12:08:11.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74402025-02-28T12:08:11.167ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74412025-02-28T12:08:11.167ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74422025-02-28T12:08:11.171ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74432025-02-28T12:08:11.171ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74442025-02-28T12:08:11.173ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74452025-02-28T12:08:11.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74462025-02-28T12:08:11.180ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 721.894576ms iteration = 19 status = {}
74472025-02-28T12:08:14.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 321 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = f907b638-8eb5-449a-b364-b6ae180709f1 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
74482025-02-28T12:08:20.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 56 reason = Timeout
74492025-02-28T12:08:20.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74502025-02-28T12:08:20.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74512025-02-28T12:08:20.459ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 56 reason = Timeout
74522025-02-28T12:08:20.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74532025-02-28T12:08:20.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74542025-02-28T12:08:20.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74552025-02-28T12:08:20.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
74562025-02-28T12:08:20.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74572025-02-28T12:08:20.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74582025-02-28T12:08:20.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74592025-02-28T12:08:20.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74602025-02-28T12:08:20.468ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74612025-02-28T12:08:20.470ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
74622025-02-28T12:08:20.470ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74632025-02-28T12:08:20.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74642025-02-28T12:08:20.484ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
74652025-02-28T12:08:20.484ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 27.460363ms iteration = 56 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
74662025-02-28T12:08:20.491ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74672025-02-28T12:08:20.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74682025-02-28T12:08:20.528ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 69.015546ms iteration = 56 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
74692025-02-28T12:08:24.254ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 000fe079-a696-4ba7-906b-2d1b30bbd28a roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
74702025-02-28T12:08:24.254ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 000fe079-a696-4ba7-906b-2d1b30bbd28a resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
74712025-02-28T12:08:24.257ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3197 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 000fe079-a696-4ba7-906b-2d1b30bbd28a response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
74722025-02-28T12:08:24.257ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 5db1ac08-6e2a-4613-a310-b3913765bf85 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
74732025-02-28T12:08:24.257ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 5db1ac08-6e2a-4613-a310-b3913765bf85 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
74742025-02-28T12:08:24.259ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2271 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 5db1ac08-6e2a-4613-a310-b3913765bf85 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
74752025-02-28T12:08:24.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 239 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 4e860770-e92e-4498-8298-11c375c0c426 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
74762025-02-28T12:08:30.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 57 reason = Timeout
74772025-02-28T12:08:30.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74782025-02-28T12:08:30.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74792025-02-28T12:08:30.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 57 reason = Timeout
74802025-02-28T12:08:30.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74812025-02-28T12:08:30.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74822025-02-28T12:08:30.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74832025-02-28T12:08:30.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74842025-02-28T12:08:30.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74852025-02-28T12:08:30.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
74862025-02-28T12:08:30.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74872025-02-28T12:08:30.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74882025-02-28T12:08:30.462ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.257803ms iteration = 57 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
74892025-02-28T12:08:30.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74902025-02-28T12:08:30.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74912025-02-28T12:08:30.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74922025-02-28T12:08:30.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
74932025-02-28T12:08:30.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74942025-02-28T12:08:30.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74952025-02-28T12:08:30.489ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
74962025-02-28T12:08:30.489ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 33.033437ms iteration = 57 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
74972025-02-28T12:08:34.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 252 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 01d7e101-b2df-4832-80a3-9309773b3f53 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
74982025-02-28T12:08:39.253ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 6bf5739d-28ae-4210-966b-4eca0df5d35c roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
74992025-02-28T12:08:39.253ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 6bf5739d-28ae-4210-966b-4eca0df5d35c resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
75002025-02-28T12:08:39.256ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2980 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = 6bf5739d-28ae-4210-966b-4eca0df5d35c response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
75012025-02-28T12:08:39.256ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = e9f67857-5cfa-4769-9a9d-e54d8a850ead roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
75022025-02-28T12:08:39.256ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = e9f67857-5cfa-4769-9a9d-e54d8a850ead resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
75032025-02-28T12:08:39.258ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2377 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62104 req_id = e9f67857-5cfa-4769-9a9d-e54d8a850ead response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
75042025-02-28T12:08:40.453ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 20 reason = Timeout
75052025-02-28T12:08:40.454ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
75062025-02-28T12:08:40.454ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 20 reason = Timeout
75072025-02-28T12:08:40.454ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
75082025-02-28T12:08:40.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
75092025-02-28T12:08:40.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
75102025-02-28T12:08:40.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 58 reason = Timeout
75112025-02-28T12:08:40.456ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 20 reason = Timeout
75122025-02-28T12:08:40.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75132025-02-28T12:08:40.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75142025-02-28T12:08:40.456ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1355, tv_nsec: 93465409 } })
75152025-02-28T12:08:40.457ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1355, tv_nsec: 93685544 } })
75162025-02-28T12:08:40.457ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
75172025-02-28T12:08:40.457ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
75182025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75192025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 58 reason = Timeout
75202025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
75212025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 20 reason = Timeout
75222025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 21 reason = Timeout
75232025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
75242025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 20 reason = Timeout
75252025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
75262025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75272025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 20 reason = Timeout
75282025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
75292025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 20 reason = Timeout
75302025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
75312025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
75322025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 20 reason = Timeout
75332025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
75342025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 20 reason = Timeout
75352025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
75362025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 20 reason = Timeout
75372025-02-28T12:08:40.458ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
75382025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75392025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 20 reason = Timeout
75402025-02-28T12:08:40.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
75412025-02-28T12:08:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
75422025-02-28T12:08:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75432025-02-28T12:08:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
75442025-02-28T12:08:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
75452025-02-28T12:08:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
75462025-02-28T12:08:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
75472025-02-28T12:08:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75482025-02-28T12:08:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
75492025-02-28T12:08:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
75502025-02-28T12:08:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
75512025-02-28T12:08:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 20 reason = Timeout
75522025-02-28T12:08:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
75532025-02-28T12:08:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 20 reason = Timeout
75542025-02-28T12:08:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
75552025-02-28T12:08:40.460ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 20 reason = Timeout
75562025-02-28T12:08:40.460ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
75572025-02-28T12:08:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 20 reason = Timeout
75582025-02-28T12:08:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
75592025-02-28T12:08:40.460ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 20 reason = Timeout
75602025-02-28T12:08:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
75612025-02-28T12:08:40.460ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.651138ms iteration = 20 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
75622025-02-28T12:08:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
75632025-02-28T12:08:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
75642025-02-28T12:08:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
75652025-02-28T12:08:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
75662025-02-28T12:08:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
75672025-02-28T12:08:40.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
75682025-02-28T12:08:40.461ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75692025-02-28T12:08:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
75702025-02-28T12:08:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
75712025-02-28T12:08:40.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75722025-02-28T12:08:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75732025-02-28T12:08:40.462ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
75742025-02-28T12:08:40.462ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
75752025-02-28T12:08:40.462ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
75762025-02-28T12:08:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
75772025-02-28T12:08:40.463ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75782025-02-28T12:08:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
75792025-02-28T12:08:40.463ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75802025-02-28T12:08:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75812025-02-28T12:08:40.464ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75822025-02-28T12:08:40.465ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75832025-02-28T12:08:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
75842025-02-28T12:08:40.465ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
75852025-02-28T12:08:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
75862025-02-28T12:08:40.466ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75872025-02-28T12:08:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
75882025-02-28T12:08:40.467ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
75892025-02-28T12:08:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
75902025-02-28T12:08:40.467ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75912025-02-28T12:08:40.468ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
75922025-02-28T12:08:40.468ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75932025-02-28T12:08:40.468ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
75942025-02-28T12:08:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
75952025-02-28T12:08:40.468ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
75962025-02-28T12:08:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75972025-02-28T12:08:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
75982025-02-28T12:08:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
75992025-02-28T12:08:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
76002025-02-28T12:08:40.469ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
76012025-02-28T12:08:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
76022025-02-28T12:08:40.471ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 12.799339ms iteration = 58 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
76032025-02-28T12:08:40.472ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76042025-02-28T12:08:40.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76052025-02-28T12:08:40.473ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76062025-02-28T12:08:40.474ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
76072025-02-28T12:08:40.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76082025-02-28T12:08:40.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
76092025-02-28T12:08:40.476ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76102025-02-28T12:08:40.477ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1355, tv_nsec: 113956023 } })
76112025-02-28T12:08:40.477ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
76122025-02-28T12:08:40.480ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
76132025-02-28T12:08:40.480ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76142025-02-28T12:08:40.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76152025-02-28T12:08:40.496ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
76162025-02-28T12:08:40.496ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 40.512458ms iteration = 58 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
76172025-02-28T12:08:40.502ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76182025-02-28T12:08:40.503ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76192025-02-28T12:08:40.508ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 48.929761ms iteration = 20 status = {"errors":[],"finish_invoked_ok":[]}
76202025-02-28T12:08:40.509ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 49.084574ms iteration = 20 status = {"errors":[],"requests_created_ok":[]}
76212025-02-28T12:08:40.510ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76222025-02-28T12:08:40.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76232025-02-28T12:08:40.515ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 55.192982ms iteration = 20 status = {"errors":[],"garbage_collect_requested":[]}
76242025-02-28T12:08:40.515ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76252025-02-28T12:08:40.515ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76262025-02-28T12:08:40.518ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
76272025-02-28T12:08:40.518ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
76282025-02-28T12:08:40.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
76292025-02-28T12:08:40.518ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
76302025-02-28T12:08:40.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
76312025-02-28T12:08:40.519ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
76322025-02-28T12:08:40.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
76332025-02-28T12:08:40.520ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
76342025-02-28T12:08:40.520ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76352025-02-28T12:08:40.520ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76362025-02-28T12:08:40.520ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
76372025-02-28T12:08:40.520ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
76382025-02-28T12:08:40.520ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
76392025-02-28T12:08:40.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76402025-02-28T12:08:40.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
76412025-02-28T12:08:40.521ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 62.353886ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
76422025-02-28T12:08:40.521ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
76432025-02-28T12:08:40.521ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
76442025-02-28T12:08:40.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
76452025-02-28T12:08:40.523ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 64.268921ms iteration = 20 status = 0
76462025-02-28T12:08:40.525ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
76472025-02-28T12:08:40.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
76482025-02-28T12:08:40.528ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
76492025-02-28T12:08:40.528ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 69.202081ms iteration = 20 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
76502025-02-28T12:08:40.537ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76512025-02-28T12:08:40.540ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
76522025-02-28T12:08:40.540ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
76532025-02-28T12:08:40.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
76542025-02-28T12:08:40.544ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
76552025-02-28T12:08:40.544ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
76562025-02-28T12:08:40.545ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5026267-cb40-4ac3-89d0-76ccd1c44620", "content-length": "26", "date": "Fri, 28 Feb 2025 12:08:40 GMT"} })
76572025-02-28T12:08:40.545ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
76582025-02-28T12:08:40.546ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
76592025-02-28T12:08:40.549ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
76602025-02-28T12:08:40.549ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
76612025-02-28T12:08:40.549ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ded1de9-9aeb-4958-9b38-e612b1b0b1cf", "content-length": "26", "date": "Fri, 28 Feb 2025 12:08:40 GMT"} })
76622025-02-28T12:08:40.549ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
76632025-02-28T12:08:40.551ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
76642025-02-28T12:08:40.559ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
76652025-02-28T12:08:40.559ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
76662025-02-28T12:08:40.559ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 100.24942ms iteration = 20 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
76672025-02-28T12:08:40.559ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
76682025-02-28T12:08:40.560ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
76692025-02-28T12:08:40.563ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 103.442328ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
76702025-02-28T12:08:40.566ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
76712025-02-28T12:08:40.566ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 107.648849ms iteration = 20 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
76722025-02-28T12:08:40.566ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
76732025-02-28T12:08:40.566ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
76742025-02-28T12:08:40.567ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6a36388-41bf-4ae6-a299-392f272553ef", "content-length": "26", "date": "Fri, 28 Feb 2025 12:08:40 GMT"} })
76752025-02-28T12:08:40.567ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
76762025-02-28T12:08:40.569ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
76772025-02-28T12:08:40.578ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
76782025-02-28T12:08:40.578ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
76792025-02-28T12:08:40.604ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 144.651452ms iteration = 20 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
76802025-02-28T12:08:40.647ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 193.814408ms iteration = 20 status = {}
76812025-02-28T12:08:40.674ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
76822025-02-28T12:08:40.675ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "33a1b1a9-c81d-4775-9676-eb7df45b8c73", "content-length": "1", "date": "Fri, 28 Feb 2025 12:08:39 GMT"} })
76832025-02-28T12:08:40.677ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
76842025-02-28T12:08:40.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
76852025-02-28T12:08:40.680ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 226.367254ms iteration = 20 status = null
76862025-02-28T12:08:40.689ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 230.180876ms iteration = 21 status = {"physical_disks_added":0}
76872025-02-28T12:08:40.707ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
76882025-02-28T12:08:40.707ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
76892025-02-28T12:08:40.708ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "50dc84c6-6971-4351-8a85-199e1ad9a5f6", "content-length": "372", "date": "Fri, 28 Feb 2025 12:08:40 GMT"} })
76902025-02-28T12:08:40.708ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
76912025-02-28T12:08:40.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
76922025-02-28T12:08:40.718ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
76932025-02-28T12:08:40.718ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
76942025-02-28T12:08:40.720ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76952025-02-28T12:08:40.723ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
76962025-02-28T12:08:40.723ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
76972025-02-28T12:08:40.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
76982025-02-28T12:08:40.725ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76992025-02-28T12:08:40.728ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
77002025-02-28T12:08:40.728ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
77012025-02-28T12:08:40.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
77022025-02-28T12:08:40.779ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
77032025-02-28T12:08:40.779ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
77042025-02-28T12:08:40.779ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "4153b3d8-7057-48d0-b8f3-11e658f576a7", "date": "Fri, 28 Feb 2025 12:08:40 GMT"} })
77052025-02-28T12:08:40.779ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77062025-02-28T12:08:40.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77072025-02-28T12:08:40.781ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77082025-02-28T12:08:40.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77092025-02-28T12:08:40.785ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77102025-02-28T12:08:40.785ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77112025-02-28T12:08:40.789ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
77122025-02-28T12:08:40.789ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
77132025-02-28T12:08:40.789ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "3f0211e2-ec74-46ce-ac4f-df3990b301aa", "date": "Fri, 28 Feb 2025 12:08:40 GMT"} })
77142025-02-28T12:08:40.791ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 332.464365ms iteration = 20 status = {}
77152025-02-28T12:08:40.794ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77162025-02-28T12:08:40.795ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77172025-02-28T12:08:40.796ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
77182025-02-28T12:08:40.796ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77192025-02-28T12:08:40.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77202025-02-28T12:08:40.799ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
77212025-02-28T12:08:40.800ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "e87cf564-ffc3-422e-99a0-a80d70866f21", "content-length": "691", "date": "Fri, 28 Feb 2025 12:08:40 GMT"} })
77222025-02-28T12:08:40.800ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
77232025-02-28T12:08:40.800ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
77242025-02-28T12:08:40.800ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
77252025-02-28T12:08:40.801ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 342.618681ms iteration = 20 status = {}
77262025-02-28T12:08:40.811ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
77272025-02-28T12:08:40.811ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "7824ee24-acc2-4a3e-b5fe-efdf2cfb61a6", "content-length": "134", "date": "Fri, 28 Feb 2025 12:08:39 GMT"} })
77282025-02-28T12:08:40.811ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
77292025-02-28T12:08:40.811ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
77302025-02-28T12:08:40.811ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
77312025-02-28T12:08:40.811ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
77322025-02-28T12:08:40.812ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77332025-02-28T12:08:40.812ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77342025-02-28T12:08:40.812ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77352025-02-28T12:08:40.812ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
77362025-02-28T12:08:40.812ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77372025-02-28T12:08:40.812ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
77382025-02-28T12:08:40.812ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
77392025-02-28T12:08:40.812ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77402025-02-28T12:08:40.812ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77412025-02-28T12:08:40.817ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
77422025-02-28T12:08:40.817ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "77b48aee-4c4f-420b-acb3-923b64127131", "content-length": "2", "date": "Fri, 28 Feb 2025 12:08:39 GMT"} })
77432025-02-28T12:08:40.817ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
77442025-02-28T12:08:40.818ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "e670d715-1ab4-446a-8465-daa1f014c894", "content-length": "51", "date": "Fri, 28 Feb 2025 12:08:39 GMT"} })
77452025-02-28T12:08:40.818ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
77462025-02-28T12:08:40.818ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
77472025-02-28T12:08:40.944ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "9abf2e58-dfa4-422f-91ca-04f08bd8b47b", "date": "Fri, 28 Feb 2025 12:08:40 GMT"} })
77482025-02-28T12:08:40.944ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
77492025-02-28T12:08:40.944ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
77502025-02-28T12:08:40.945ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbff0bce-8e42-43de-ab89-d563a2160fc6", "content-length": "682", "date": "Fri, 28 Feb 2025 12:08:40 GMT"} })
77512025-02-28T12:08:40.945ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77522025-02-28T12:08:40.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77532025-02-28T12:08:40.949ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77542025-02-28T12:08:40.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77552025-02-28T12:08:40.954ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77562025-02-28T12:08:40.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77572025-02-28T12:08:40.957ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77582025-02-28T12:08:40.957ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77592025-02-28T12:08:40.965ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 506.299682ms iteration = 20 status = {}
77602025-02-28T12:08:44.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 267 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = d0b699fb-ff59-4da1-93a5-4e3ef732b531 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
77612025-02-28T12:08:50.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 59 reason = Timeout
77622025-02-28T12:08:50.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77632025-02-28T12:08:50.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77642025-02-28T12:08:50.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 59 reason = Timeout
77652025-02-28T12:08:50.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77662025-02-28T12:08:50.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77672025-02-28T12:08:50.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
77682025-02-28T12:08:50.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
77692025-02-28T12:08:50.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77702025-02-28T12:08:50.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77712025-02-28T12:08:50.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77722025-02-28T12:08:50.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77732025-02-28T12:08:50.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77742025-02-28T12:08:50.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
77752025-02-28T12:08:50.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77762025-02-28T12:08:50.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77772025-02-28T12:08:50.481ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
77782025-02-28T12:08:50.481ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.543276ms iteration = 59 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
77792025-02-28T12:08:50.491ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77802025-02-28T12:08:50.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
77812025-02-28T12:08:50.525ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 66.844094ms iteration = 59 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
77822025-02-28T12:08:54.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 255 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = ad331ba5-0bab-49b6-9730-885460676c9e response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
77832025-02-28T12:09:00.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 60 reason = Timeout
77842025-02-28T12:09:00.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77852025-02-28T12:09:00.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77862025-02-28T12:09:00.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 60 reason = Timeout
77872025-02-28T12:09:00.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77882025-02-28T12:09:00.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77892025-02-28T12:09:00.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
77902025-02-28T12:09:00.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77912025-02-28T12:09:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
77922025-02-28T12:09:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
77932025-02-28T12:09:00.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77942025-02-28T12:09:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77952025-02-28T12:09:00.461ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.192777ms iteration = 60 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
77962025-02-28T12:09:00.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77972025-02-28T12:09:00.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77982025-02-28T12:09:00.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77992025-02-28T12:09:00.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
78002025-02-28T12:09:00.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78012025-02-28T12:09:00.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78022025-02-28T12:09:00.475ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
78032025-02-28T12:09:00.475ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.695866ms iteration = 60 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
78042025-02-28T12:09:04.261ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 292 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 7fc2abc6-ecd1-41d2-b919-8d8f5a65cd52 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
78052025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 11 reason = Timeout
78062025-02-28T12:09:10.453ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-28 11:59:10.453075100 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
78072025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 13 reason = Timeout
78082025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
78092025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 21 reason = Timeout
78102025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
78112025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 11 reason = Timeout
78122025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 11 reason = Timeout
78132025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
78142025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
78152025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
78162025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 11 reason = Timeout
78172025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 11 reason = Timeout
78182025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
78192025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
78202025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 11 reason = Timeout
78212025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 21 reason = Timeout
78222025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
78232025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 13 reason = Timeout
78242025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
78252025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
78262025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
78272025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
78282025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
78292025-02-28T12:09:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
78302025-02-28T12:09:10.454ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5ebff2d9-48b6-4fe6-b852-684bb14b4955.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bddfb0f9-1d7b-45e4-b3a5-35aa6792956b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1385, tv_nsec: 90695132 } })
78312025-02-28T12:09:10.454ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5cefabf1-6831-46ff-98eb-1a3b5abad8b9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1385, tv_nsec: 90832855 } })
78322025-02-28T12:09:10.454ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
78332025-02-28T12:09:10.454ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.534937ms iteration = 11 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
78342025-02-28T12:09:10.454ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
78352025-02-28T12:09:10.454ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.557118ms iteration = 11 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
78362025-02-28T12:09:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
78372025-02-28T12:09:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
78382025-02-28T12:09:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
78392025-02-28T12:09:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
78402025-02-28T12:09:10.456ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-28 11:59:10.453075100 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
78412025-02-28T12:09:10.456ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.418503ms iteration = 11 status = {"expiration":"2025-02-28T11:59:10.453075100Z","pruned":[]}
78422025-02-28T12:09:10.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 61 reason = Timeout
78432025-02-28T12:09:10.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 12 reason = Timeout
78442025-02-28T12:09:10.456ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
78452025-02-28T12:09:10.456ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 5.64\u{b5}s iteration = 12 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
78462025-02-28T12:09:10.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78472025-02-28T12:09:10.456ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 21 reason = Timeout
78482025-02-28T12:09:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78492025-02-28T12:09:10.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 11 reason = Timeout
78502025-02-28T12:09:10.457ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 21 reason = Timeout
78512025-02-28T12:09:10.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
78522025-02-28T12:09:10.457ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
78532025-02-28T12:09:10.457ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 21 reason = Timeout
78542025-02-28T12:09:10.457ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
78552025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 11 reason = Timeout
78562025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
78572025-02-28T12:09:10.458ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
78582025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 21 reason = Timeout
78592025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
78602025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78612025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 21 reason = Timeout
78622025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
78632025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Timeout
78642025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 22 reason = Timeout
78652025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78662025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
78672025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 21 reason = Timeout
78682025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
78692025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 4 reason = Timeout
78702025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
78712025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 3 reason = Timeout
78722025-02-28T12:09:10.458ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
78732025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
78742025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 21 reason = Timeout
78752025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
78762025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 61 reason = Timeout
78772025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
78782025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 21 reason = Timeout
78792025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
78802025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 21 reason = Timeout
78812025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
78822025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
78832025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78842025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
78852025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
78862025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
78872025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78882025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
78892025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78902025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
78912025-02-28T12:09:10.458ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1385, tv_nsec: 95218221 } })
78922025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
78932025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
78942025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
78952025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
78962025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
78972025-02-28T12:09:10.458ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1385, tv_nsec: 95314973 } })
78982025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
78992025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
79002025-02-28T12:09:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
79012025-02-28T12:09:10.458ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
79022025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 21 reason = Timeout
79032025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 3 reason = Timeout
79042025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
79052025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 21 reason = Timeout
79062025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
79072025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
79082025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 21 reason = Timeout
79092025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
79102025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 21 reason = Timeout
79112025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
79122025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 21 reason = Timeout
79132025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
79142025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 11 reason = Timeout
79152025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79162025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
79172025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 11 reason = Timeout
79182025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
79192025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): activating background_task = saga_recovery iteration = 2 reason = Timeout
79202025-02-28T12:09:10.459ZDEBGnexus (SagaRecoverer): listing candidate sagas for recovery
79212025-02-28T12:09:10.459ZDEBGnexus (SagaRecoverer): roles roles = RoleSet { roles: {} }
79222025-02-28T12:09:10.459ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
79232025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
79242025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
79252025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
79262025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
79272025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
79282025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
79292025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
79302025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
79312025-02-28T12:09:10.459ZDEBGnexus (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
79322025-02-28T12:09:10.459ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79332025-02-28T12:09:10.460ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
79342025-02-28T12:09:10.460ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
79352025-02-28T12:09:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
79362025-02-28T12:09:10.462ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.816571ms iteration = 21 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
79372025-02-28T12:09:10.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
79382025-02-28T12:09:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
79392025-02-28T12:09:10.464ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79402025-02-28T12:09:10.465ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79412025-02-28T12:09:10.467ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79422025-02-28T12:09:10.468ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79432025-02-28T12:09:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
79442025-02-28T12:09:10.468ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79452025-02-28T12:09:10.468ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
79462025-02-28T12:09:10.468ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
79472025-02-28T12:09:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79482025-02-28T12:09:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
79492025-02-28T12:09:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
79502025-02-28T12:09:10.469ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
79512025-02-28T12:09:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
79522025-02-28T12:09:10.469ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79532025-02-28T12:09:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
79542025-02-28T12:09:10.470ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
79552025-02-28T12:09:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
79562025-02-28T12:09:10.470ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79572025-02-28T12:09:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
79582025-02-28T12:09:10.472ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
79592025-02-28T12:09:10.472ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
79602025-02-28T12:09:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
79612025-02-28T12:09:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
79622025-02-28T12:09:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
79632025-02-28T12:09:10.473ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
79642025-02-28T12:09:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
79652025-02-28T12:09:10.475ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79662025-02-28T12:09:10.476ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79672025-02-28T12:09:10.476ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
79682025-02-28T12:09:10.476ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
79692025-02-28T12:09:10.476ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
79702025-02-28T12:09:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
79712025-02-28T12:09:10.477ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 18.031386ms iteration = 21 status = {"errors":[],"finish_invoked_ok":[]}
79722025-02-28T12:09:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
79732025-02-28T12:09:10.482ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
79742025-02-28T12:09:10.482ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
79752025-02-28T12:09:10.482ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
79762025-02-28T12:09:10.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
79772025-02-28T12:09:10.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
79782025-02-28T12:09:10.485ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79792025-02-28T12:09:10.487ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 28.055787ms iteration = 11 status = {"errors":[],"found_port_ok":[]}
79802025-02-28T12:09:10.487ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
79812025-02-28T12:09:10.487ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
79822025-02-28T12:09:10.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
79832025-02-28T12:09:10.490ZDEBGnexus (BackgroundTasks): inventory_prune_one: eligible for removal background_task = inventory_collection candidates = [(629d8169-3b39-4c37-bfde-5c8ee7e6e15c, 0), (cc5b0957-282b-42ba-a76f-75dc60602319, 0), (04093fd7-6ac9-46ad-91b7-2e450a0b3469, 0), (15665a6a-6930-47e8-ae90-d9ea47eeca82, 0)] collection_id = 629d8169-3b39-4c37-bfde-5c8ee7e6e15c
79842025-02-28T12:09:10.490ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
79852025-02-28T12:09:10.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
79862025-02-28T12:09:10.493ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1385, tv_nsec: 129788255 } })
79872025-02-28T12:09:10.493ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
79882025-02-28T12:09:10.493ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
79892025-02-28T12:09:10.493ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
79902025-02-28T12:09:10.498ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
79912025-02-28T12:09:10.499ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
79922025-02-28T12:09:10.502ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
79932025-02-28T12:09:10.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
79942025-02-28T12:09:10.510ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79952025-02-28T12:09:10.515ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
79962025-02-28T12:09:10.515ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
79972025-02-28T12:09:10.515ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
79982025-02-28T12:09:10.515ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
79992025-02-28T12:09:10.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
80002025-02-28T12:09:10.515ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
80012025-02-28T12:09:10.515ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
80022025-02-28T12:09:10.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
80032025-02-28T12:09:10.516ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
80042025-02-28T12:09:10.516ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
80052025-02-28T12:09:10.516ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
80062025-02-28T12:09:10.516ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
80072025-02-28T12:09:10.516ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80082025-02-28T12:09:10.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
80092025-02-28T12:09:10.518ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80102025-02-28T12:09:10.518ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
80112025-02-28T12:09:10.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
80122025-02-28T12:09:10.519ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
80132025-02-28T12:09:10.519ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
80142025-02-28T12:09:10.519ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
80152025-02-28T12:09:10.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
80162025-02-28T12:09:10.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
80172025-02-28T12:09:10.520ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
80182025-02-28T12:09:10.520ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
80192025-02-28T12:09:10.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
80202025-02-28T12:09:10.522ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
80212025-02-28T12:09:10.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
80222025-02-28T12:09:10.523ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
80232025-02-28T12:09:10.523ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80242025-02-28T12:09:10.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
80252025-02-28T12:09:10.525ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
80262025-02-28T12:09:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
80272025-02-28T12:09:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
80282025-02-28T12:09:10.528ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
80292025-02-28T12:09:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
80302025-02-28T12:09:10.530ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 71.346182ms iteration = 21 status = {"errors":[],"requests_created_ok":[]}
80312025-02-28T12:09:10.531ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80322025-02-28T12:09:10.531ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80332025-02-28T12:09:10.531ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
80342025-02-28T12:09:10.531ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
80352025-02-28T12:09:10.531ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 72.462619ms iteration = 11 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
80362025-02-28T12:09:10.532ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
80372025-02-28T12:09:10.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
80382025-02-28T12:09:10.532ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80392025-02-28T12:09:10.534ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
80402025-02-28T12:09:10.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
80412025-02-28T12:09:10.535ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
80422025-02-28T12:09:10.535ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
80432025-02-28T12:09:10.536ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 77.116172ms iteration = 21 status = {"errors":[],"garbage_collect_requested":[]}
80442025-02-28T12:09:10.536ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
80452025-02-28T12:09:10.536ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
80462025-02-28T12:09:10.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
80472025-02-28T12:09:10.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
80482025-02-28T12:09:10.537ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
80492025-02-28T12:09:10.537ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
80502025-02-28T12:09:10.538ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
80512025-02-28T12:09:10.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
80522025-02-28T12:09:10.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
80532025-02-28T12:09:10.539ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80542025-02-28T12:09:10.539ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
80552025-02-28T12:09:10.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
80562025-02-28T12:09:10.540ZINFOnexus (SagaRecoverer): listed in-progress sagas count = 0 file = nexus/src/app/background/tasks/saga_recovery.rs:438
80572025-02-28T12:09:10.540ZDEBGnexus (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 81.690432ms iteration = 2 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
80582025-02-28T12:09:10.540ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 82.946362ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
80592025-02-28T12:09:10.542ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
80602025-02-28T12:09:10.542ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80612025-02-28T12:09:10.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80622025-02-28T12:09:10.543ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 85.649638ms iteration = 22 status = {"physical_disks_added":0}
80632025-02-28T12:09:10.548ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80642025-02-28T12:09:10.550ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
80652025-02-28T12:09:10.550ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
80662025-02-28T12:09:10.550ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 92.052022ms iteration = 11 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
80672025-02-28T12:09:10.550ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
80682025-02-28T12:09:10.550ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
80692025-02-28T12:09:10.551ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
80702025-02-28T12:09:10.551ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
80712025-02-28T12:09:10.551ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
80722025-02-28T12:09:10.553ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
80732025-02-28T12:09:10.553ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
80742025-02-28T12:09:10.553ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 0 version = 2
80752025-02-28T12:09:10.553ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
80762025-02-28T12:09:10.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
80772025-02-28T12:09:10.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
80782025-02-28T12:09:10.554ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
80792025-02-28T12:09:10.554ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
80802025-02-28T12:09:10.554ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
80812025-02-28T12:09:10.554ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80822025-02-28T12:09:10.554ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
80832025-02-28T12:09:10.556ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
80842025-02-28T12:09:10.556ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
80852025-02-28T12:09:10.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
80862025-02-28T12:09:10.558ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
80872025-02-28T12:09:10.558ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 100.477736ms iteration = 21 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
80882025-02-28T12:09:10.558ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80892025-02-28T12:09:10.562ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
80902025-02-28T12:09:10.562ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
80912025-02-28T12:09:10.562ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
80922025-02-28T12:09:10.563ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
80932025-02-28T12:09:10.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
80942025-02-28T12:09:10.564ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
80952025-02-28T12:09:10.564ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
80962025-02-28T12:09:10.565ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
80972025-02-28T12:09:10.565ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
80982025-02-28T12:09:10.567ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80992025-02-28T12:09:10.568ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
81002025-02-28T12:09:10.568ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
81012025-02-28T12:09:10.570ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
81022025-02-28T12:09:10.570ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
81032025-02-28T12:09:10.570ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
81042025-02-28T12:09:10.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
81052025-02-28T12:09:10.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
81062025-02-28T12:09:10.571ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
81072025-02-28T12:09:10.571ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
81082025-02-28T12:09:10.571ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 113.663224ms iteration = 11 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
81092025-02-28T12:09:10.571ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 114.861943ms iteration = 61 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
81102025-02-28T12:09:10.572ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
81112025-02-28T12:09:10.572ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 114.881874ms iteration = 21 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
81122025-02-28T12:09:10.573ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
81132025-02-28T12:09:10.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
81142025-02-28T12:09:10.575ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81152025-02-28T12:09:10.579ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
81162025-02-28T12:09:10.579ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
81172025-02-28T12:09:10.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
81182025-02-28T12:09:10.582ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 124.603028ms iteration = 21 status = 0
81192025-02-28T12:09:10.591ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 133.493013ms iteration = 61 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
81202025-02-28T12:09:10.592ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
81212025-02-28T12:09:10.593ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
81222025-02-28T12:09:10.597ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81232025-02-28T12:09:10.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
81242025-02-28T12:09:10.604ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
81252025-02-28T12:09:10.604ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
81262025-02-28T12:09:10.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
81272025-02-28T12:09:10.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
81282025-02-28T12:09:10.607ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 1 version = 2
81292025-02-28T12:09:10.607ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
81302025-02-28T12:09:10.607ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-02-28 11:59:52.542241 UTC dns_group = external generation = 2
81312025-02-28T12:09:10.607ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 154.308705ms iteration = 11 status = {"generation":2}
81322025-02-28T12:09:10.608ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
81332025-02-28T12:09:10.608ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
81342025-02-28T12:09:10.608ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
81352025-02-28T12:09:10.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
81362025-02-28T12:09:10.610ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
81372025-02-28T12:09:10.610ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
81382025-02-28T12:09:10.613ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d25f2c47-3ff7-4bf7-b928-2f36172a0e43", "content-length": "26", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
81392025-02-28T12:09:10.613ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
81402025-02-28T12:09:10.614ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 155.203786ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
81412025-02-28T12:09:10.615ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
81422025-02-28T12:09:10.616ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
81432025-02-28T12:09:10.616ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
81442025-02-28T12:09:10.616ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 158.441884ms iteration = 21 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
81452025-02-28T12:09:10.617ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
81462025-02-28T12:09:10.617ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81472025-02-28T12:09:10.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
81482025-02-28T12:09:10.620ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81492025-02-28T12:09:10.623ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-28T11:59:08.719399Z, time_modified: 2025-02-28T11:59:08.719399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }) }
81502025-02-28T12:09:10.623ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 169.963863ms iteration = 11 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]}},"default_endpoint":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
81512025-02-28T12:09:10.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81522025-02-28T12:09:10.623ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
81532025-02-28T12:09:10.623ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
81542025-02-28T12:09:10.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
81552025-02-28T12:09:10.626ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
81562025-02-28T12:09:10.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
81572025-02-28T12:09:10.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
81582025-02-28T12:09:10.626ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
81592025-02-28T12:09:10.627ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
81602025-02-28T12:09:10.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
81612025-02-28T12:09:10.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81622025-02-28T12:09:10.631ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
81632025-02-28T12:09:10.631ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
81642025-02-28T12:09:10.633ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
81652025-02-28T12:09:10.633ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
81662025-02-28T12:09:10.633ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-02-28 11:59:51.915187 UTC dns_group = internal generation = 1
81672025-02-28T12:09:10.633ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 180.551998ms iteration = 11 status = {"generation":1}
81682025-02-28T12:09:10.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
81692025-02-28T12:09:10.640ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
81702025-02-28T12:09:10.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
81712025-02-28T12:09:10.642ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
81722025-02-28T12:09:10.643ZINFOnexus (BackgroundTasks): removed inventory collection background_task = inventory_collection collection_id = 629d8169-3b39-4c37-bfde-5c8ee7e6e15c file = nexus/db-queries/src/db/datastore/inventory.rs:1442 ncabooses = 12 nclickhouse_keeper_membership = 0 ncollections = 1 ndatasets = 0 nerrors = 0 nnics = 0 nnvme_disk_firmware = 0 nphysical_disks = 0 nrot_pages = 8 nrots = 2 nsled_agent_zones = 0 nsled_agents = 0 nsps = 2 nzones = 0 nzpools = 0
81732025-02-28T12:09:10.643ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81742025-02-28T12:09:10.643ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
81752025-02-28T12:09:10.643ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
81762025-02-28T12:09:10.643ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81772025-02-28T12:09:10.644ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
81782025-02-28T12:09:10.646ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81792025-02-28T12:09:10.649ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
81802025-02-28T12:09:10.650ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
81812025-02-28T12:09:10.650ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
81822025-02-28T12:09:10.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
81832025-02-28T12:09:10.651ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
81842025-02-28T12:09:10.654ZDEBGnexus (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(cc5b0957-282b-42ba-a76f-75dc60602319, 0), (04093fd7-6ac9-46ad-91b7-2e450a0b3469, 0), (15665a6a-6930-47e8-ae90-d9ea47eeca82, 0)]
81852025-02-28T12:09:10.655ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1385, tv_nsec: 291859908 } })
81862025-02-28T12:09:10.672ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
81872025-02-28T12:09:10.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
81882025-02-28T12:09:10.685ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 226.20031ms iteration = 21 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
81892025-02-28T12:09:10.769ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
81902025-02-28T12:09:10.769ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
81912025-02-28T12:09:10.770ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "431f0a3b-e594-4570-a0db-3148548d0e7a", "content-length": "26", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
81922025-02-28T12:09:10.770ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
81932025-02-28T12:09:10.785ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
81942025-02-28T12:09:10.785ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
81952025-02-28T12:09:10.785ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff9f09a2-68d7-4b2c-b898-4afa436daf23", "content-length": "26", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
81962025-02-28T12:09:10.785ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
81972025-02-28T12:09:10.786ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
81982025-02-28T12:09:10.788ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
81992025-02-28T12:09:10.917ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
82002025-02-28T12:09:10.917ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
82012025-02-28T12:09:10.917ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e9ebe7a-c877-44f3-b348-89d21f7aeb6d", "content-length": "372", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82022025-02-28T12:09:10.917ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
82032025-02-28T12:09:10.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
82042025-02-28T12:09:10.932ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
82052025-02-28T12:09:10.932ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "c0ff4dcd-1945-4963-b564-4ef4767f93fc", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
82062025-02-28T12:09:10.932ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
82072025-02-28T12:09:10.934ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "8948364c-d0fb-4742-a348-af454a54078c", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
82082025-02-28T12:09:10.934ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
82092025-02-28T12:09:10.935ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
82102025-02-28T12:09:10.935ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 482.847656ms iteration = 13 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
82112025-02-28T12:09:10.953ZDEBGnexus (BackgroundTasks): begin collection background_task = inventory_collection
82122025-02-28T12:09:10.953ZDEBGnexus (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[fd00:1122:3344:101::2]:12225
82132025-02-28T12:09:10.953ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/ignition
82142025-02-28T12:09:10.953ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0268929-1256-4091-ac8d-31dd6b8076d8", "content-length": "476", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82152025-02-28T12:09:10.953ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0
82162025-02-28T12:09:10.954ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6ab1dec-4af6-402b-8527-77e04297d091", "content-length": "734", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82172025-02-28T12:09:10.954ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0
82182025-02-28T12:09:10.954ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b48920d-0fdc-43c0-ac0a-e6e9dcf67540", "content-length": "111", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82192025-02-28T12:09:10.954ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1
82202025-02-28T12:09:10.954ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "15b01253-4c73-4bf2-96c9-3128273e4a13", "content-length": "111", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82212025-02-28T12:09:10.954ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0
82222025-02-28T12:09:10.955ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a11f6ff2-e5aa-4420-8e09-031eafe234c7", "content-length": "167", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82232025-02-28T12:09:10.955ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1
82242025-02-28T12:09:10.955ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "857ecb8a-184b-4c4d-83a9-6c71a330d4db", "content-length": "167", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82252025-02-28T12:09:10.955ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0
82262025-02-28T12:09:10.955ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dea12607-2b98-40e4-95fe-a77338c2a7cf", "content-length": "175", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82272025-02-28T12:09:10.955ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1
82282025-02-28T12:09:10.956ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbc41f50-3778-4e67-ae8b-15eae1a6e219", "content-length": "175", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82292025-02-28T12:09:10.956ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa
82302025-02-28T12:09:10.956ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d8880ef-4267-4ffb-9269-13d4a8fb0a33", "content-length": "702", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82312025-02-28T12:09:10.956ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
82322025-02-28T12:09:10.956ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92dd7514-541c-4023-8a4d-7e7ed7dcd212", "content-length": "727", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82332025-02-28T12:09:10.956ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
82342025-02-28T12:09:10.956ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d22287bb-ddbf-4134-885b-124ed22af533", "content-length": "729", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82352025-02-28T12:09:10.956ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
82362025-02-28T12:09:10.956ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "827b1296-71ce-4d49-8286-fbf8be7a1380", "content-length": "728", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82372025-02-28T12:09:10.956ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0
82382025-02-28T12:09:10.956ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12152eb4-6d5d-4c6a-82dd-c541a23724c8", "content-length": "722", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82392025-02-28T12:09:10.956ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0
82402025-02-28T12:09:10.957ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a60bac9-52ec-4e0a-ad10-553281723a4b", "content-length": "109", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82412025-02-28T12:09:10.957ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1
82422025-02-28T12:09:10.957ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a520da86-22a9-4e7d-b2bc-af1b47c7c7d2", "content-length": "109", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82432025-02-28T12:09:10.957ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0
82442025-02-28T12:09:10.957ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e42e8f6-52a1-4592-8939-9e87001ff360", "content-length": "169", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82452025-02-28T12:09:10.957ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1
82462025-02-28T12:09:10.957ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "454923b7-2459-42ea-99bc-0b27ac86ce67", "content-length": "169", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82472025-02-28T12:09:10.958ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0
82482025-02-28T12:09:10.958ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "777eaf75-fa55-4922-8330-ec620697b0f6", "content-length": "178", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82492025-02-28T12:09:10.958ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1
82502025-02-28T12:09:10.958ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "206c93ee-5bd7-4d3e-8c4a-3a774bff96e3", "content-length": "178", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82512025-02-28T12:09:10.958ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa
82522025-02-28T12:09:10.958ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f7d0685-8ee6-4102-bf77-4c3be30b1353", "content-length": "702", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82532025-02-28T12:09:10.958ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
82542025-02-28T12:09:10.958ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02a5fc12-76b7-4162-869c-3d7a38039b9e", "content-length": "727", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82552025-02-28T12:09:10.958ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
82562025-02-28T12:09:10.958ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8cc04a9-b19a-4937-bb63-9f425a7f96a1", "content-length": "729", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82572025-02-28T12:09:10.958ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
82582025-02-28T12:09:10.959ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57a98a8f-98be-4aa6-9e69-a619c06ec8d8", "content-length": "728", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82592025-02-28T12:09:10.959ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82602025-02-28T12:09:10.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82612025-02-28T12:09:10.964ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 511.427246ms iteration = 21 status = {}
82622025-02-28T12:09:10.966ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
82632025-02-28T12:09:10.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
82642025-02-28T12:09:10.968ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82652025-02-28T12:09:10.970ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
82662025-02-28T12:09:10.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
82672025-02-28T12:09:10.970ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
82682025-02-28T12:09:10.970ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
82692025-02-28T12:09:10.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
82702025-02-28T12:09:10.973ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82712025-02-28T12:09:10.974ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
82722025-02-28T12:09:10.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
82732025-02-28T12:09:10.975ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
82742025-02-28T12:09:10.975ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
82752025-02-28T12:09:10.975ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82762025-02-28T12:09:10.975ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
82772025-02-28T12:09:10.975ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 517.738168ms iteration = 4 status = {"blueprint_id":"7a676565-49af-418f-9d1f-e6c070e71120","inventory_collection_id":"15665a6a-6930-47e8-ae90-d9ea47eeca82","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
82782025-02-28T12:09:10.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82792025-02-28T12:09:10.976ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
82802025-02-28T12:09:10.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
82812025-02-28T12:09:10.977ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82822025-02-28T12:09:10.978ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82832025-02-28T12:09:10.979ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
82842025-02-28T12:09:10.979ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82852025-02-28T12:09:10.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82862025-02-28T12:09:10.983ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
82872025-02-28T12:09:10.983ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
82882025-02-28T12:09:10.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
82892025-02-28T12:09:10.984ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
82902025-02-28T12:09:10.984ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "0db68d23-5aad-4028-bc38-faea35327487", "content-length": "1", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
82912025-02-28T12:09:10.985ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
82922025-02-28T12:09:10.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
82932025-02-28T12:09:10.986ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
82942025-02-28T12:09:10.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
82952025-02-28T12:09:10.988ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 534.924273ms iteration = 21 status = null
82962025-02-28T12:09:10.988ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
82972025-02-28T12:09:10.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
82982025-02-28T12:09:10.990ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
82992025-02-28T12:09:10.990ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
83002025-02-28T12:09:10.990ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "95b6762e-778e-460d-a03f-1ba6dedf180d", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
83012025-02-28T12:09:10.990ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83022025-02-28T12:09:10.990ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83032025-02-28T12:09:10.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83042025-02-28T12:09:10.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83052025-02-28T12:09:10.993ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83062025-02-28T12:09:10.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83072025-02-28T12:09:10.993ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83082025-02-28T12:09:10.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83092025-02-28T12:09:10.995ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83102025-02-28T12:09:10.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83112025-02-28T12:09:10.996ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83122025-02-28T12:09:10.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83132025-02-28T12:09:10.997ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83142025-02-28T12:09:10.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83152025-02-28T12:09:11.000ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
83162025-02-28T12:09:11.000ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
83172025-02-28T12:09:11.000ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "6e490065-4e48-4542-8c4f-de2e25612bbd", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
83182025-02-28T12:09:11.003ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 545.345855ms iteration = 21 status = {}
83192025-02-28T12:09:11.022ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83202025-02-28T12:09:11.024ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
83212025-02-28T12:09:11.024ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
83222025-02-28T12:09:11.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
83232025-02-28T12:09:11.027ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
83242025-02-28T12:09:11.027ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "57627b71-8637-4ad1-813e-d251f71ef92a", "date": "Fri, 28 Feb 2025 12:09:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
83252025-02-28T12:09:11.027ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
83262025-02-28T12:09:11.027ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
83272025-02-28T12:09:11.028ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a8752e5-f10a-4231-96d8-14ab37fd27c8", "content-length": "2", "date": "Fri, 28 Feb 2025 12:09:11 GMT"} })
83282025-02-28T12:09:11.028ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:619 sled = http://[fd00:1122:3344:101::1]:12345
83292025-02-28T12:09:11.030ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "8f12a7aa-c5b3-4a5b-82cc-1315ddbb1ceb", "date": "Fri, 28 Feb 2025 12:09:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
83302025-02-28T12:09:11.030ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
83312025-02-28T12:09:11.031ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 572.260584ms iteration = 3 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":2,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"operation":"list","target_sled":"50ac5f18-4748-43d4-a4ac-8d2114194efc","time":"2025-02-28T12:04:11.359459081Z"},{"operation":"list","target_sled":"50ac5f18-4748-43d4-a4ac-8d2114194efc","time":"2025-02-28T12:09:11.028643484Z"}]}
83322025-02-28T12:09:11.032ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "789823a4-7a65-467d-be1c-839dab6409f4", "date": "Fri, 28 Feb 2025 12:09:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
83332025-02-28T12:09:11.032ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
83342025-02-28T12:09:11.035ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
83352025-02-28T12:09:11.035ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 582.226235ms iteration = 13 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
83362025-02-28T12:09:11.036ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83372025-02-28T12:09:11.041ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
83382025-02-28T12:09:11.041ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83392025-02-28T12:09:11.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83402025-02-28T12:09:11.043ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83412025-02-28T12:09:11.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83422025-02-28T12:09:11.045ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83432025-02-28T12:09:11.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83442025-02-28T12:09:11.047ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83452025-02-28T12:09:11.050ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
83462025-02-28T12:09:11.050ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83472025-02-28T12:09:11.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83482025-02-28T12:09:11.055ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83492025-02-28T12:09:11.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83502025-02-28T12:09:11.075ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
83512025-02-28T12:09:11.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
83522025-02-28T12:09:11.113ZDEBGnexus (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[fd00:1122:3344:101::1]:12345
83532025-02-28T12:09:11.113ZDEBGnexus (BackgroundTasks): client request SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/inventory
83542025-02-28T12:09:11.118ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83552025-02-28T12:09:11.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83562025-02-28T12:09:11.120ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83572025-02-28T12:09:11.120ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
83582025-02-28T12:09:11.120ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83592025-02-28T12:09:11.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83602025-02-28T12:09:11.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83612025-02-28T12:09:11.122ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
83622025-02-28T12:09:11.122ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
83632025-02-28T12:09:11.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
83642025-02-28T12:09:11.124ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83652025-02-28T12:09:11.124ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
83662025-02-28T12:09:11.124ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
83672025-02-28T12:09:11.124ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83682025-02-28T12:09:11.124ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83692025-02-28T12:09:11.125ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83702025-02-28T12:09:11.127ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
83712025-02-28T12:09:11.127ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
83722025-02-28T12:09:11.127ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
83732025-02-28T12:09:11.128ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83742025-02-28T12:09:11.128ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83752025-02-28T12:09:11.131ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83762025-02-28T12:09:11.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83772025-02-28T12:09:11.134ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83782025-02-28T12:09:11.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83792025-02-28T12:09:11.136ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83802025-02-28T12:09:11.140ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
83812025-02-28T12:09:11.140ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83822025-02-28T12:09:11.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83832025-02-28T12:09:11.142ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83842025-02-28T12:09:11.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83852025-02-28T12:09:11.144ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
83862025-02-28T12:09:11.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
83872025-02-28T12:09:11.144ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
83882025-02-28T12:09:11.145ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "92f84827-b7fa-4c4d-a874-8c8c7ff71c88", "content-length": "134", "date": "Fri, 28 Feb 2025 12:09:11 GMT"} })
83892025-02-28T12:09:11.145ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
83902025-02-28T12:09:11.145ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
83912025-02-28T12:09:11.145ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
83922025-02-28T12:09:11.145ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
83932025-02-28T12:09:11.145ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
83942025-02-28T12:09:11.145ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
83952025-02-28T12:09:11.145ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
83962025-02-28T12:09:11.145ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
83972025-02-28T12:09:11.145ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83982025-02-28T12:09:11.146ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
83992025-02-28T12:09:11.146ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
84002025-02-28T12:09:11.146ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
84012025-02-28T12:09:11.146ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
84022025-02-28T12:09:11.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
84032025-02-28T12:09:11.149ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
84042025-02-28T12:09:11.149ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c79fe45-6fdc-482f-9745-c87080e2540a", "content-length": "2", "date": "Fri, 28 Feb 2025 12:09:11 GMT"} })
84052025-02-28T12:09:11.149ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
84062025-02-28T12:09:11.149ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9123a53-3889-420d-80a2-fd31879fcbf6", "content-length": "51", "date": "Fri, 28 Feb 2025 12:09:11 GMT"} })
84072025-02-28T12:09:11.149ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
84082025-02-28T12:09:11.149ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
84092025-02-28T12:09:11.150ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
84102025-02-28T12:09:11.150ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
84112025-02-28T12:09:11.150ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
84122025-02-28T12:09:11.157ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: efece645-4fc7-4f5a-900b-9b0d58342421, kind: Service { id: db281835-e35c-480d-b1a2-07f321816d55 }, name: Name("external-dns-db281835-e35c-480d-b1a2-07f321816d55"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 88])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fb9b46af-fe90-4b4d-b831-9e9a193e0996, kind: Service { id: 5cefabf1-6831-46ff-98eb-1a3b5abad8b9 }, name: Name("external-dns-5cefabf1-6831-46ff-98eb-1a3b5abad8b9"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 243, 164])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 08731947-b4fa-4741-bf20-8ff14850f21a, kind: Service { id: 3bbfd7d6-3f72-4034-ad35-e956e1c12923 }, name: Name("nexus-3bbfd7d6-3f72-4034-ad35-e956e1c12923"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 4814ec6a-93e1-40fa-ba73-651e41bc6ee3, kind: Service { id: c1b34bc5-1c2f-4d3c-808c-b7a441aac295 }, name: Name("nexus-c1b34bc5-1c2f-4d3c-808c-b7a441aac295"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 152])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e56f48b3-247c-430d-8e79-57098c59bab5, kind: Service { id: e594478d-1d06-4692-a5d7-cded9c00f6d4 }, name: Name("nexus-e594478d-1d06-4692-a5d7-cded9c00f6d4"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 182, 65])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
84132025-02-28T12:09:11.157ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
84142025-02-28T12:09:11.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
84152025-02-28T12:09:11.159ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84162025-02-28T12:09:11.161ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
84172025-02-28T12:09:11.161ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
84182025-02-28T12:09:11.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
84192025-02-28T12:09:11.163ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
84202025-02-28T12:09:11.163ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: efece645-4fc7-4f5a-900b-9b0d58342421, kind: Service { id: db281835-e35c-480d-b1a2-07f321816d55 }, name: Name("external-dns-db281835-e35c-480d-b1a2-07f321816d55"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 88])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fb9b46af-fe90-4b4d-b831-9e9a193e0996, kind: Service { id: 5cefabf1-6831-46ff-98eb-1a3b5abad8b9 }, name: Name("external-dns-5cefabf1-6831-46ff-98eb-1a3b5abad8b9"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 243, 164])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 08731947-b4fa-4741-bf20-8ff14850f21a, kind: Service { id: 3bbfd7d6-3f72-4034-ad35-e956e1c12923 }, name: Name("nexus-3bbfd7d6-3f72-4034-ad35-e956e1c12923"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 4814ec6a-93e1-40fa-ba73-651e41bc6ee3, kind: Service { id: c1b34bc5-1c2f-4d3c-808c-b7a441aac295 }, name: Name("nexus-c1b34bc5-1c2f-4d3c-808c-b7a441aac295"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 152])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e56f48b3-247c-430d-8e79-57098c59bab5, kind: Service { id: e594478d-1d06-4692-a5d7-cded9c00f6d4 }, name: Name("nexus-e594478d-1d06-4692-a5d7-cded9c00f6d4"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 182, 65])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
84212025-02-28T12:09:11.163ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
84222025-02-28T12:09:11.255ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
84232025-02-28T12:09:11.258ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "afc76aa5-9195-4f53-be55-2598675431b1", "content-length": "691", "date": "Fri, 28 Feb 2025 12:09:11 GMT"} })
84242025-02-28T12:09:11.258ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
84252025-02-28T12:09:11.258ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
84262025-02-28T12:09:11.258ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
84272025-02-28T12:09:11.259ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 801.735694ms iteration = 21 status = {}
84282025-02-28T12:09:11.302ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "d0b02291-4865-4a26-b2d9-9d160c9dd68f", "date": "Fri, 28 Feb 2025 12:09:11 GMT"} })
84292025-02-28T12:09:11.302ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
84302025-02-28T12:09:11.302ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
84312025-02-28T12:09:11.306ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "2626e1f5-fc09-4fad-92fd-c384160f5b8b", "content-length": "682", "date": "Fri, 28 Feb 2025 12:09:11 GMT"} })
84322025-02-28T12:09:11.306ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
84332025-02-28T12:09:11.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
84342025-02-28T12:09:11.314ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 50ac5f18-4748-43d4-a4ac-8d2114194efc, time_created: 2025-02-28T11:59:46.871334Z, time_modified: 2025-02-28T11:59:46.871334Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
84352025-02-28T12:09:11.314ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
84362025-02-28T12:09:11.314ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
84372025-02-28T12:09:11.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
84382025-02-28T12:09:11.317ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
84392025-02-28T12:09:11.317ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
84402025-02-28T12:09:11.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
84412025-02-28T12:09:11.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
84422025-02-28T12:09:11.319ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84432025-02-28T12:09:11.322ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
84442025-02-28T12:09:11.322ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
84452025-02-28T12:09:11.322ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
84462025-02-28T12:09:11.324ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
84472025-02-28T12:09:11.325ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
84482025-02-28T12:09:11.332ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 874.989023ms iteration = 21 status = {}
84492025-02-28T12:09:11.354ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
84502025-02-28T12:09:11.372ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "62549617-ae68-4399-acad-28b5214e3b45", "date": "Fri, 28 Feb 2025 12:09:10 GMT"} })
84512025-02-28T12:09:11.374ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
84522025-02-28T12:09:11.374ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 916.067474ms
84532025-02-28T12:09:11.374ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 916.076374ms iteration = 3 status = {}
84542025-02-28T12:09:11.547ZDEBGnexus (BackgroundTasks): client response SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "33b8e7c0-b87c-43d7-8186-5b442e7e4460", "content-length": "31973", "date": "Fri, 28 Feb 2025 12:09:11 GMT"} })
84552025-02-28T12:09:11.573ZDEBGnexus (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
84562025-02-28T12:09:11.573ZDEBGnexus (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
84572025-02-28T12:09:11.573ZDEBGnexus (BackgroundTasks): finished collection background_task = inventory_collection
84582025-02-28T12:09:11.574ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84592025-02-28T12:09:11.575ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84602025-02-28T12:09:11.576ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84612025-02-28T12:09:11.578ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
84622025-02-28T12:09:11.578ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84632025-02-28T12:09:11.578ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84642025-02-28T12:09:11.750ZINFOnexus (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 2148ba0b-50f4-43ef-8b17-b2f556d999c3 file = nexus/db-queries/src/db/datastore/inventory.rs:1010
84652025-02-28T12:09:11.750ZDEBGnexus (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 2148ba0b-50f4-43ef-8b17-b2f556d999c3 time_started = 2025-02-28 12:09:10.953490 UTC
84662025-02-28T12:09:11.750ZDEBGnexus (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.292649226s iteration = 2 status = {"collection_id":"2148ba0b-50f4-43ef-8b17-b2f556d999c3","time_done":"2025-02-28 12:09:11.573110 UTC","time_started":"2025-02-28 12:09:10.953490 UTC"}
84672025-02-28T12:09:11.750ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 5 reason = Dependency
84682025-02-28T12:09:11.750ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 23 reason = Dependency
84692025-02-28T12:09:11.750ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
84702025-02-28T12:09:11.750ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
84712025-02-28T12:09:11.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
84722025-02-28T12:09:11.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
84732025-02-28T12:09:11.752ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84742025-02-28T12:09:11.753ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84752025-02-28T12:09:11.755ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
84762025-02-28T12:09:11.755ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
84772025-02-28T12:09:11.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
84782025-02-28T12:09:11.755ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
84792025-02-28T12:09:11.755ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
84802025-02-28T12:09:11.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
84812025-02-28T12:09:11.758ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
84822025-02-28T12:09:11.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
84832025-02-28T12:09:11.760ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84842025-02-28T12:09:11.762ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
84852025-02-28T12:09:11.762ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
84862025-02-28T12:09:11.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
84872025-02-28T12:09:11.808ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 57.801005ms iteration = 23 status = {"physical_disks_added":0}
84882025-02-28T12:09:11.899ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
84892025-02-28T12:09:11.900ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
84902025-02-28T12:09:11.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
84912025-02-28T12:09:11.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
84922025-02-28T12:09:11.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84932025-02-28T12:09:11.930ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
84942025-02-28T12:09:11.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
84952025-02-28T12:09:11.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
84962025-02-28T12:09:11.932ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84972025-02-28T12:09:11.934ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
84982025-02-28T12:09:11.934ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
84992025-02-28T12:09:11.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
85002025-02-28T12:09:11.937ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
85012025-02-28T12:09:11.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
85022025-02-28T12:09:11.939ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85032025-02-28T12:09:11.940ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
85042025-02-28T12:09:11.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
85052025-02-28T12:09:11.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
85062025-02-28T12:09:11.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85072025-02-28T12:09:11.943ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
85082025-02-28T12:09:11.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
85092025-02-28T12:09:11.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
85102025-02-28T12:09:11.945ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
85112025-02-28T12:09:11.946ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 195.441888ms iteration = 5 status = {"blueprint_id":"7a676565-49af-418f-9d1f-e6c070e71120","inventory_collection_id":"a84bcc24-1946-49c1-8d43-7345b4933ff0","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
85122025-02-28T12:09:14.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 258 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 8eef5518-1fe7-4407-adf7-c8e701eefdb8 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
85132025-02-28T12:09:20.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 62 reason = Timeout
85142025-02-28T12:09:20.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85152025-02-28T12:09:20.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85162025-02-28T12:09:20.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 62 reason = Timeout
85172025-02-28T12:09:20.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85182025-02-28T12:09:20.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85192025-02-28T12:09:20.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85202025-02-28T12:09:20.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
85212025-02-28T12:09:20.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85222025-02-28T12:09:20.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85232025-02-28T12:09:20.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85242025-02-28T12:09:20.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85252025-02-28T12:09:20.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85262025-02-28T12:09:20.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
85272025-02-28T12:09:20.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85282025-02-28T12:09:20.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85292025-02-28T12:09:20.478ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
85302025-02-28T12:09:20.478ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.179256ms iteration = 62 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
85312025-02-28T12:09:20.488ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85322025-02-28T12:09:20.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85332025-02-28T12:09:20.516ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 58.525323ms iteration = 62 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
85342025-02-28T12:09:24.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 297 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 441f2c94-53c1-4d45-abfd-571e68309759 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
85352025-02-28T12:09:30.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 63 reason = Timeout
85362025-02-28T12:09:30.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85372025-02-28T12:09:30.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85382025-02-28T12:09:30.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 63 reason = Timeout
85392025-02-28T12:09:30.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85402025-02-28T12:09:30.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85412025-02-28T12:09:30.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85422025-02-28T12:09:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
85432025-02-28T12:09:30.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85442025-02-28T12:09:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85452025-02-28T12:09:30.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85462025-02-28T12:09:30.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85472025-02-28T12:09:30.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85482025-02-28T12:09:30.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
85492025-02-28T12:09:30.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85502025-02-28T12:09:30.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85512025-02-28T12:09:30.484ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
85522025-02-28T12:09:30.484ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 28.467517ms iteration = 63 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
85532025-02-28T12:09:30.490ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85542025-02-28T12:09:30.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85552025-02-28T12:09:30.492ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.698417ms iteration = 63 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
85562025-02-28T12:09:34.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 288 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 418e724d-5236-4954-b80f-6ae1b81b140b response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
85572025-02-28T12:09:39.253ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::d]:50417
85582025-02-28T12:09:39.253ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50417 req_id = 82c18c3e-d400-4e7a-8773-c2dd256695d3 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
85592025-02-28T12:09:39.253ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50417 req_id = 82c18c3e-d400-4e7a-8773-c2dd256695d3 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
85602025-02-28T12:09:39.256ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2945 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50417 req_id = 82c18c3e-d400-4e7a-8773-c2dd256695d3 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
85612025-02-28T12:09:39.256ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50417 req_id = d35f3cf0-d24f-45ce-9384-9b87501acffe roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
85622025-02-28T12:09:39.256ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50417 req_id = d35f3cf0-d24f-45ce-9384-9b87501acffe resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
85632025-02-28T12:09:39.258ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2221 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50417 req_id = d35f3cf0-d24f-45ce-9384-9b87501acffe response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
85642025-02-28T12:09:40.452ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 22 reason = Timeout
85652025-02-28T12:09:40.453ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
85662025-02-28T12:09:40.453ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 22 reason = Timeout
85672025-02-28T12:09:40.453ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
85682025-02-28T12:09:40.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
85692025-02-28T12:09:40.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
85702025-02-28T12:09:40.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 64 reason = Timeout
85712025-02-28T12:09:40.455ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 22 reason = Timeout
85722025-02-28T12:09:40.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85732025-02-28T12:09:40.456ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1415, tv_nsec: 92930116 } })
85742025-02-28T12:09:40.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85752025-02-28T12:09:40.456ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
85762025-02-28T12:09:40.456ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1415, tv_nsec: 93497000 } })
85772025-02-28T12:09:40.456ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
85782025-02-28T12:09:40.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85792025-02-28T12:09:40.457ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 22 reason = Timeout
85802025-02-28T12:09:40.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 22 reason = Timeout
85812025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
85822025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
85832025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 22 reason = Timeout
85842025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
85852025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 22 reason = Timeout
85862025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 22 reason = Timeout
85872025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
85882025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85892025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 64 reason = Timeout
85902025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85912025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 22 reason = Timeout
85922025-02-28T12:09:40.458ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
85932025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
85942025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 22 reason = Timeout
85952025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
85962025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
85972025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 22 reason = Timeout
85982025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
85992025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 24 reason = Timeout
86002025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
86012025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
86022025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
86032025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86042025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
86052025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
86062025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
86072025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
86082025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
86092025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
86102025-02-28T12:09:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
86112025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 22 reason = Timeout
86122025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 22 reason = Timeout
86132025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
86142025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
86152025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 22 reason = Timeout
86162025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
86172025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 22 reason = Timeout
86182025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
86192025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 22 reason = Timeout
86202025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
86212025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
86222025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
86232025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
86242025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
86252025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.542526ms iteration = 22 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
86262025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
86272025-02-28T12:09:40.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86282025-02-28T12:09:40.460ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86292025-02-28T12:09:40.460ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86302025-02-28T12:09:40.461ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86312025-02-28T12:09:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
86322025-02-28T12:09:40.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
86332025-02-28T12:09:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
86342025-02-28T12:09:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
86352025-02-28T12:09:40.462ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
86362025-02-28T12:09:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
86372025-02-28T12:09:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
86382025-02-28T12:09:40.463ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86392025-02-28T12:09:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
86402025-02-28T12:09:40.463ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
86412025-02-28T12:09:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86422025-02-28T12:09:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
86432025-02-28T12:09:40.463ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
86442025-02-28T12:09:40.463ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
86452025-02-28T12:09:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
86462025-02-28T12:09:40.464ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
86472025-02-28T12:09:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
86482025-02-28T12:09:40.465ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86492025-02-28T12:09:40.466ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86502025-02-28T12:09:40.466ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
86512025-02-28T12:09:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
86522025-02-28T12:09:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
86532025-02-28T12:09:40.467ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
86542025-02-28T12:09:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
86552025-02-28T12:09:40.467ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
86562025-02-28T12:09:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
86572025-02-28T12:09:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
86582025-02-28T12:09:40.469ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 10.083814ms iteration = 22 status = {"errors":[],"garbage_collect_requested":[]}
86592025-02-28T12:09:40.471ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
86602025-02-28T12:09:40.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
86612025-02-28T12:09:40.475ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86622025-02-28T12:09:40.476ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
86632025-02-28T12:09:40.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
86642025-02-28T12:09:40.477ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
86652025-02-28T12:09:40.477ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
86662025-02-28T12:09:40.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
86672025-02-28T12:09:40.479ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1415, tv_nsec: 115977562 } })
86682025-02-28T12:09:40.479ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
86692025-02-28T12:09:40.479ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
86702025-02-28T12:09:40.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
86712025-02-28T12:09:40.490ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
86722025-02-28T12:09:40.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
86732025-02-28T12:09:40.510ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
86742025-02-28T12:09:40.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
86752025-02-28T12:09:40.511ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 52.227191ms iteration = 22 status = {"errors":[],"requests_created_ok":[]}
86762025-02-28T12:09:40.514ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
86772025-02-28T12:09:40.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
86782025-02-28T12:09:40.515ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86792025-02-28T12:09:40.519ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 61.30264ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
86802025-02-28T12:09:40.520ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
86812025-02-28T12:09:40.520ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
86822025-02-28T12:09:40.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
86832025-02-28T12:09:40.522ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 64.988559ms iteration = 22 status = 0
86842025-02-28T12:09:40.525ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86852025-02-28T12:09:40.528ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
86862025-02-28T12:09:40.528ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
86872025-02-28T12:09:40.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
86882025-02-28T12:09:40.528ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
86892025-02-28T12:09:40.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
86902025-02-28T12:09:40.528ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
86912025-02-28T12:09:40.528ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
86922025-02-28T12:09:40.528ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
86932025-02-28T12:09:40.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
86942025-02-28T12:09:40.531ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
86952025-02-28T12:09:40.531ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 73.442863ms iteration = 22 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
86962025-02-28T12:09:40.531ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
86972025-02-28T12:09:40.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
86982025-02-28T12:09:40.537ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 78.384482ms iteration = 22 status = {"errors":[],"finish_invoked_ok":[]}
86992025-02-28T12:09:40.538ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 80.37566ms iteration = 64 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
87002025-02-28T12:09:40.539ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87012025-02-28T12:09:40.539ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
87022025-02-28T12:09:40.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
87032025-02-28T12:09:40.541ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87042025-02-28T12:09:40.542ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
87052025-02-28T12:09:40.542ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87062025-02-28T12:09:40.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87072025-02-28T12:09:40.543ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
87082025-02-28T12:09:40.543ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
87092025-02-28T12:09:40.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
87102025-02-28T12:09:40.555ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
87112025-02-28T12:09:40.555ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 100.021555ms iteration = 64 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
87122025-02-28T12:09:40.556ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
87132025-02-28T12:09:40.556ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
87142025-02-28T12:09:40.557ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "10cd07f7-0071-404b-90b2-6d425aece5e6", "content-length": "26", "date": "Fri, 28 Feb 2025 12:09:40 GMT"} })
87152025-02-28T12:09:40.557ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
87162025-02-28T12:09:40.559ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
87172025-02-28T12:09:40.561ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 103.798156ms iteration = 24 status = {"physical_disks_added":0}
87182025-02-28T12:09:40.562ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
87192025-02-28T12:09:40.562ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
87202025-02-28T12:09:40.562ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
87212025-02-28T12:09:40.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
87222025-02-28T12:09:40.563ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 104.265377ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
87232025-02-28T12:09:40.564ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
87242025-02-28T12:09:40.564ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
87252025-02-28T12:09:40.583ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
87262025-02-28T12:09:40.583ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 125.065459ms iteration = 22 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
87272025-02-28T12:09:40.612ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
87282025-02-28T12:09:40.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
87292025-02-28T12:09:40.612ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
87302025-02-28T12:09:40.612ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
87312025-02-28T12:09:40.612ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 154.795526ms iteration = 22 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
87322025-02-28T12:09:40.614ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87332025-02-28T12:09:40.616ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
87342025-02-28T12:09:40.616ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
87352025-02-28T12:09:40.616ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 157.267986ms iteration = 22 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
87362025-02-28T12:09:40.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
87372025-02-28T12:09:40.618ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87382025-02-28T12:09:40.619ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
87392025-02-28T12:09:40.619ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
87402025-02-28T12:09:40.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
87412025-02-28T12:09:40.662ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
87422025-02-28T12:09:40.662ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
87432025-02-28T12:09:40.662ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ec5f988-93f3-46a8-9870-d659385e791f", "content-length": "26", "date": "Fri, 28 Feb 2025 12:09:40 GMT"} })
87442025-02-28T12:09:40.662ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
87452025-02-28T12:09:40.665ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
87462025-02-28T12:09:40.683ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
87472025-02-28T12:09:40.683ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
87482025-02-28T12:09:40.683ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1bdc4c1-0053-4a43-ab0f-04107f9f1902", "content-length": "26", "date": "Fri, 28 Feb 2025 12:09:40 GMT"} })
87492025-02-28T12:09:40.683ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
87502025-02-28T12:09:40.685ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
87512025-02-28T12:09:40.706ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
87522025-02-28T12:09:40.706ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
87532025-02-28T12:09:40.707ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "4915d258-0453-4848-bbe0-b9634a97fa93", "content-length": "372", "date": "Fri, 28 Feb 2025 12:09:40 GMT"} })
87542025-02-28T12:09:40.707ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
87552025-02-28T12:09:40.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
87562025-02-28T12:09:40.719ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 266.78172ms iteration = 22 status = {}
87572025-02-28T12:09:40.724ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
87582025-02-28T12:09:40.724ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
87592025-02-28T12:09:40.724ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "ed9bdcf8-5ebb-416d-b9ef-f388c1936337", "date": "Fri, 28 Feb 2025 12:09:40 GMT"} })
87602025-02-28T12:09:40.724ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
87612025-02-28T12:09:40.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
87622025-02-28T12:09:40.727ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
87632025-02-28T12:09:40.727ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
87642025-02-28T12:09:40.730ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
87652025-02-28T12:09:40.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
87662025-02-28T12:09:40.753ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
87672025-02-28T12:09:40.753ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "87301860-0d05-4905-b946-77446761fd96", "content-length": "691", "date": "Fri, 28 Feb 2025 12:09:40 GMT"} })
87682025-02-28T12:09:40.754ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
87692025-02-28T12:09:40.754ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
87702025-02-28T12:09:40.754ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
87712025-02-28T12:09:40.755ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 297.328937ms iteration = 22 status = {}
87722025-02-28T12:09:40.757ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
87732025-02-28T12:09:40.757ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
87742025-02-28T12:09:40.758ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "a1715cc2-0d10-4b67-bd11-9a940216b41a", "date": "Fri, 28 Feb 2025 12:09:40 GMT"} })
87752025-02-28T12:09:40.760ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 302.182335ms iteration = 22 status = {}
87762025-02-28T12:09:40.847ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
87772025-02-28T12:09:40.847ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd5a6ae7-4692-41b9-a10b-b203316a4e8d", "content-length": "1", "date": "Fri, 28 Feb 2025 12:09:40 GMT"} })
87782025-02-28T12:09:40.849ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
87792025-02-28T12:09:40.850ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
87802025-02-28T12:09:40.851ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 398.793647ms iteration = 22 status = null
87812025-02-28T12:09:40.911ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
87822025-02-28T12:09:40.911ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
87832025-02-28T12:09:40.913ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
87842025-02-28T12:09:40.913ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
87852025-02-28T12:09:40.913ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
87862025-02-28T12:09:40.924ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
87872025-02-28T12:09:40.924ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "974f3dba-cc48-4311-883c-2193f7ed0cc7", "content-length": "134", "date": "Fri, 28 Feb 2025 12:09:40 GMT"} })
87882025-02-28T12:09:40.924ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
87892025-02-28T12:09:40.924ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
87902025-02-28T12:09:40.925ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
87912025-02-28T12:09:40.925ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
87922025-02-28T12:09:40.925ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
87932025-02-28T12:09:40.925ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
87942025-02-28T12:09:40.925ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
87952025-02-28T12:09:40.925ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
87962025-02-28T12:09:40.925ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
87972025-02-28T12:09:40.925ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
87982025-02-28T12:09:40.925ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
87992025-02-28T12:09:40.925ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88002025-02-28T12:09:40.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88012025-02-28T12:09:40.927ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
88022025-02-28T12:09:40.928ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "01d55de6-e5b3-4df7-8dea-17489b35b473", "content-length": "2", "date": "Fri, 28 Feb 2025 12:09:40 GMT"} })
88032025-02-28T12:09:40.928ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
88042025-02-28T12:09:40.928ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cae3144-4152-49da-81fa-e050295d23b4", "content-length": "51", "date": "Fri, 28 Feb 2025 12:09:40 GMT"} })
88052025-02-28T12:09:40.928ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
88062025-02-28T12:09:40.928ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
88072025-02-28T12:09:41.211ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "5c9a6574-608e-429d-9027-104737a6f2c4", "date": "Fri, 28 Feb 2025 12:09:40 GMT"} })
88082025-02-28T12:09:41.211ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
88092025-02-28T12:09:41.211ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
88102025-02-28T12:09:41.211ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "475baa05-3c0d-4f79-bbb3-6a408e6af9e6", "content-length": "682", "date": "Fri, 28 Feb 2025 12:09:41 GMT"} })
88112025-02-28T12:09:41.211ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88122025-02-28T12:09:41.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88132025-02-28T12:09:41.215ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88142025-02-28T12:09:41.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88152025-02-28T12:09:41.219ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88162025-02-28T12:09:41.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88172025-02-28T12:09:41.221ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88182025-02-28T12:09:41.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88192025-02-28T12:09:41.228ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 770.368818ms iteration = 22 status = {}
88202025-02-28T12:09:44.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 266 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = d4c6c013-339d-464c-95d1-1e59aba1c00d response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
88212025-02-28T12:09:50.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 65 reason = Timeout
88222025-02-28T12:09:50.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88232025-02-28T12:09:50.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88242025-02-28T12:09:50.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 65 reason = Timeout
88252025-02-28T12:09:50.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
88262025-02-28T12:09:50.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88272025-02-28T12:09:50.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
88282025-02-28T12:09:50.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
88292025-02-28T12:09:50.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
88302025-02-28T12:09:50.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
88312025-02-28T12:09:50.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88322025-02-28T12:09:50.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88332025-02-28T12:09:50.461ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.541875ms iteration = 65 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
88342025-02-28T12:09:50.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88352025-02-28T12:09:50.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88362025-02-28T12:09:50.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88372025-02-28T12:09:50.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
88382025-02-28T12:09:50.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88392025-02-28T12:09:50.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88402025-02-28T12:09:50.477ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
88412025-02-28T12:09:50.477ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.72208ms iteration = 65 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
88422025-02-28T12:09:54.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 299 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 38de32c2-e1bb-41e4-b23f-0e97cb59a781 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
88432025-02-28T12:10:00.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 66 reason = Timeout
88442025-02-28T12:10:00.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88452025-02-28T12:10:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88462025-02-28T12:10:00.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 66 reason = Timeout
88472025-02-28T12:10:00.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
88482025-02-28T12:10:00.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
88492025-02-28T12:10:00.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88502025-02-28T12:10:00.460ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
88512025-02-28T12:10:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
88522025-02-28T12:10:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
88532025-02-28T12:10:00.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88542025-02-28T12:10:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88552025-02-28T12:10:00.461ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.802992ms iteration = 66 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
88562025-02-28T12:10:00.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88572025-02-28T12:10:00.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88582025-02-28T12:10:00.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88592025-02-28T12:10:00.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
88602025-02-28T12:10:00.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88612025-02-28T12:10:00.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88622025-02-28T12:10:00.481ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
88632025-02-28T12:10:00.481ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.302461ms iteration = 66 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
88642025-02-28T12:10:04.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 270 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = fabce528-3465-45c6-a29a-04d91f5154ef response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
88652025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 12 reason = Timeout
88662025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 12 reason = Timeout
88672025-02-28T12:10:10.453ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-28 12:00:10.453468927 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
88682025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
88692025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
88702025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
88712025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 23 reason = Timeout
88722025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
88732025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 23 reason = Timeout
88742025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
88752025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 14 reason = Timeout
88762025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 14 reason = Timeout
88772025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 12 reason = Timeout
88782025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
88792025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
88802025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 12 reason = Timeout
88812025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 12 reason = Timeout
88822025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 12 reason = Timeout
88832025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
88842025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
88852025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
88862025-02-28T12:10:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
88872025-02-28T12:10:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
88882025-02-28T12:10:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
88892025-02-28T12:10:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
88902025-02-28T12:10:10.454ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5cefabf1-6831-46ff-98eb-1a3b5abad8b9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1445, tv_nsec: 91215224 } })
88912025-02-28T12:10:10.454ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5ebff2d9-48b6-4fe6-b852-684bb14b4955.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bddfb0f9-1d7b-45e4-b3a5-35aa6792956b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1445, tv_nsec: 91373468 } })
88922025-02-28T12:10:10.454ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
88932025-02-28T12:10:10.454ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 997.804\u{b5}s iteration = 12 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
88942025-02-28T12:10:10.454ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
88952025-02-28T12:10:10.454ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.205049ms iteration = 12 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
88962025-02-28T12:10:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
88972025-02-28T12:10:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
88982025-02-28T12:10:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
88992025-02-28T12:10:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
89002025-02-28T12:10:10.456ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-28 12:00:10.453468927 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
89012025-02-28T12:10:10.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 67 reason = Timeout
89022025-02-28T12:10:10.456ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 2.570162ms iteration = 12 status = {"expiration":"2025-02-28T12:00:10.453468927Z","pruned":[]}
89032025-02-28T12:10:10.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89042025-02-28T12:10:10.456ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 23 reason = Timeout
89052025-02-28T12:10:10.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 13 reason = Timeout
89062025-02-28T12:10:10.456ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
89072025-02-28T12:10:10.456ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 9.82\u{b5}s iteration = 13 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
89082025-02-28T12:10:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89092025-02-28T12:10:10.457ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1445, tv_nsec: 94526514 } })
89102025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 12 reason = Timeout
89112025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
89122025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 23 reason = Timeout
89132025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
89142025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 23 reason = Timeout
89152025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
89162025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 23 reason = Timeout
89172025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
89182025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 25 reason = Timeout
89192025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 23 reason = Timeout
89202025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
89212025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89222025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 23 reason = Timeout
89232025-02-28T12:10:10.457ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
89242025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89252025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 12 reason = Timeout
89262025-02-28T12:10:10.457ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
89272025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 23 reason = Timeout
89282025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
89292025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 23 reason = Timeout
89302025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
89312025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
89322025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
89332025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 67 reason = Timeout
89342025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
89352025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 23 reason = Timeout
89362025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
89372025-02-28T12:10:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
89382025-02-28T12:10:10.457ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
89392025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89402025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
89412025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89422025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
89432025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89442025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
89452025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
89462025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
89472025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
89482025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
89492025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
89502025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
89512025-02-28T12:10:10.458ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1445, tv_nsec: 95334864 } })
89522025-02-28T12:10:10.458ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
89532025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89542025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 23 reason = Timeout
89552025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 23 reason = Timeout
89562025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
89572025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
89582025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 12 reason = Timeout
89592025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
89602025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 23 reason = Timeout
89612025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
89622025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 23 reason = Timeout
89632025-02-28T12:10:10.458ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
89642025-02-28T12:10:10.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 23 reason = Timeout
89652025-02-28T12:10:10.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
89662025-02-28T12:10:10.459ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 12 reason = Timeout
89672025-02-28T12:10:10.459ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
89682025-02-28T12:10:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
89692025-02-28T12:10:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
89702025-02-28T12:10:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
89712025-02-28T12:10:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
89722025-02-28T12:10:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
89732025-02-28T12:10:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
89742025-02-28T12:10:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
89752025-02-28T12:10:10.459ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89762025-02-28T12:10:10.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89772025-02-28T12:10:10.461ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
89782025-02-28T12:10:10.461ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
89792025-02-28T12:10:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
89802025-02-28T12:10:10.461ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.432551ms iteration = 23 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
89812025-02-28T12:10:10.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
89822025-02-28T12:10:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
89832025-02-28T12:10:10.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89842025-02-28T12:10:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89852025-02-28T12:10:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
89862025-02-28T12:10:10.465ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89872025-02-28T12:10:10.466ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89882025-02-28T12:10:10.466ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89892025-02-28T12:10:10.468ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
89902025-02-28T12:10:10.468ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89912025-02-28T12:10:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89922025-02-28T12:10:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
89932025-02-28T12:10:10.468ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
89942025-02-28T12:10:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
89952025-02-28T12:10:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
89962025-02-28T12:10:10.469ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89972025-02-28T12:10:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89982025-02-28T12:10:10.470ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89992025-02-28T12:10:10.470ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90002025-02-28T12:10:10.473ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
90012025-02-28T12:10:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
90022025-02-28T12:10:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
90032025-02-28T12:10:10.473ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
90042025-02-28T12:10:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
90052025-02-28T12:10:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
90062025-02-28T12:10:10.473ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90072025-02-28T12:10:10.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90082025-02-28T12:10:10.475ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 16.885897ms iteration = 23 status = {"errors":[],"garbage_collect_requested":[]}
90092025-02-28T12:10:10.481ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
90102025-02-28T12:10:10.481ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
90112025-02-28T12:10:10.481ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 22.569305ms iteration = 12 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
90122025-02-28T12:10:10.482ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90132025-02-28T12:10:10.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90142025-02-28T12:10:10.485ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 26.11017ms iteration = 12 status = {"errors":[],"found_port_ok":[]}
90152025-02-28T12:10:10.485ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90162025-02-28T12:10:10.486ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
90172025-02-28T12:10:10.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
90182025-02-28T12:10:10.487ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
90192025-02-28T12:10:10.487ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
90202025-02-28T12:10:10.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
90212025-02-28T12:10:10.487ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
90222025-02-28T12:10:10.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
90232025-02-28T12:10:10.490ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90242025-02-28T12:10:10.491ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90252025-02-28T12:10:10.491ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90262025-02-28T12:10:10.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90272025-02-28T12:10:10.492ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1445, tv_nsec: 129174601 } })
90282025-02-28T12:10:10.492ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
90292025-02-28T12:10:10.493ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
90302025-02-28T12:10:10.493ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90312025-02-28T12:10:10.493ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90322025-02-28T12:10:10.493ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
90332025-02-28T12:10:10.493ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
90342025-02-28T12:10:10.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
90352025-02-28T12:10:10.495ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90362025-02-28T12:10:10.499ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
90372025-02-28T12:10:10.499ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
90382025-02-28T12:10:10.499ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
90392025-02-28T12:10:10.509ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90402025-02-28T12:10:10.511ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
90412025-02-28T12:10:10.512ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
90422025-02-28T12:10:10.512ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
90432025-02-28T12:10:10.513ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
90442025-02-28T12:10:10.513ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 57.508189ms iteration = 67 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
90452025-02-28T12:10:10.514ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
90462025-02-28T12:10:10.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
90472025-02-28T12:10:10.516ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 58.948893ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
90482025-02-28T12:10:10.517ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
90492025-02-28T12:10:10.517ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
90502025-02-28T12:10:10.517ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
90512025-02-28T12:10:10.517ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
90522025-02-28T12:10:10.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
90532025-02-28T12:10:10.518ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
90542025-02-28T12:10:10.518ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
90552025-02-28T12:10:10.518ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
90562025-02-28T12:10:10.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
90572025-02-28T12:10:10.519ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90582025-02-28T12:10:10.520ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90592025-02-28T12:10:10.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90602025-02-28T12:10:10.520ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
90612025-02-28T12:10:10.520ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
90622025-02-28T12:10:10.520ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
90632025-02-28T12:10:10.521ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
90642025-02-28T12:10:10.521ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
90652025-02-28T12:10:10.521ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
90662025-02-28T12:10:10.521ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
90672025-02-28T12:10:10.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
90682025-02-28T12:10:10.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
90692025-02-28T12:10:10.522ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.248702ms iteration = 67 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
90702025-02-28T12:10:10.522ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
90712025-02-28T12:10:10.522ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
90722025-02-28T12:10:10.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
90732025-02-28T12:10:10.524ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90742025-02-28T12:10:10.527ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
90752025-02-28T12:10:10.527ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
90762025-02-28T12:10:10.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
90772025-02-28T12:10:10.530ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 71.608039ms iteration = 23 status = {"errors":[],"finish_invoked_ok":[]}
90782025-02-28T12:10:10.531ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
90792025-02-28T12:10:10.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
90802025-02-28T12:10:10.531ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
90812025-02-28T12:10:10.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
90822025-02-28T12:10:10.532ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90832025-02-28T12:10:10.532ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
90842025-02-28T12:10:10.532ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90852025-02-28T12:10:10.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
90862025-02-28T12:10:10.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90872025-02-28T12:10:10.534ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
90882025-02-28T12:10:10.534ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
90892025-02-28T12:10:10.534ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
90902025-02-28T12:10:10.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
90912025-02-28T12:10:10.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
90922025-02-28T12:10:10.534ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
90932025-02-28T12:10:10.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
90942025-02-28T12:10:10.535ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90952025-02-28T12:10:10.535ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 76.794525ms iteration = 23 status = {"errors":[],"requests_created_ok":[]}
90962025-02-28T12:10:10.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90972025-02-28T12:10:10.536ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90982025-02-28T12:10:10.536ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 77.731946ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
90992025-02-28T12:10:10.537ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91002025-02-28T12:10:10.538ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
91012025-02-28T12:10:10.538ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
91022025-02-28T12:10:10.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
91032025-02-28T12:10:10.543ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
91042025-02-28T12:10:10.543ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
91052025-02-28T12:10:10.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
91062025-02-28T12:10:10.545ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 87.542093ms iteration = 23 status = 0
91072025-02-28T12:10:10.551ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
91082025-02-28T12:10:10.551ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
91092025-02-28T12:10:10.553ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
91102025-02-28T12:10:10.553ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 95.510925ms iteration = 23 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
91112025-02-28T12:10:10.558ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
91122025-02-28T12:10:10.558ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
91132025-02-28T12:10:10.558ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 100.24384ms iteration = 23 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
91142025-02-28T12:10:10.562ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
91152025-02-28T12:10:10.562ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
91162025-02-28T12:10:10.562ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
91172025-02-28T12:10:10.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
91182025-02-28T12:10:10.564ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
91192025-02-28T12:10:10.564ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 0 version = 2
91202025-02-28T12:10:10.564ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
91212025-02-28T12:10:10.564ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
91222025-02-28T12:10:10.564ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91232025-02-28T12:10:10.567ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
91242025-02-28T12:10:10.567ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
91252025-02-28T12:10:10.567ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
91262025-02-28T12:10:10.568ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91272025-02-28T12:10:10.568ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 109.830662ms iteration = 23 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
91282025-02-28T12:10:10.569ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
91292025-02-28T12:10:10.569ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
91302025-02-28T12:10:10.569ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 112.182938ms iteration = 12 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
91312025-02-28T12:10:10.570ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
91322025-02-28T12:10:10.570ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
91332025-02-28T12:10:10.570ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-02-28 11:59:51.915187 UTC dns_group = internal generation = 1
91342025-02-28T12:10:10.570ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 116.710178ms iteration = 12 status = {"generation":1}
91352025-02-28T12:10:10.571ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
91362025-02-28T12:10:10.571ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
91372025-02-28T12:10:10.571ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
91382025-02-28T12:10:10.571ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
91392025-02-28T12:10:10.571ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 114.119865ms iteration = 23 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
91402025-02-28T12:10:10.572ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91412025-02-28T12:10:10.573ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 1 version = 2
91422025-02-28T12:10:10.573ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
91432025-02-28T12:10:10.573ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-02-28 11:59:52.542241 UTC dns_group = external generation = 2
91442025-02-28T12:10:10.573ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 120.082849ms iteration = 12 status = {"generation":2}
91452025-02-28T12:10:10.573ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
91462025-02-28T12:10:10.573ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
91472025-02-28T12:10:10.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
91482025-02-28T12:10:10.611ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
91492025-02-28T12:10:10.611ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
91502025-02-28T12:10:10.611ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fab749a5-7ad5-4ebb-98f7-355a2d84ed8e", "content-length": "26", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} })
91512025-02-28T12:10:10.611ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
91522025-02-28T12:10:10.614ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
91532025-02-28T12:10:10.619ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
91542025-02-28T12:10:10.619ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
91552025-02-28T12:10:10.619ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "07f46790-03db-43e0-a52b-54bc790823f6", "content-length": "26", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} })
91562025-02-28T12:10:10.619ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
91572025-02-28T12:10:10.622ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
91582025-02-28T12:10:10.636ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 178.329704ms iteration = 25 status = {"physical_disks_added":0}
91592025-02-28T12:10:10.637ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
91602025-02-28T12:10:10.637ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 179.781569ms iteration = 12 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
91612025-02-28T12:10:10.640ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
91622025-02-28T12:10:10.644ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-28T11:59:08.719399Z, time_modified: 2025-02-28T11:59:08.719399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }) }
91632025-02-28T12:10:10.644ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 190.621751ms iteration = 12 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]}},"default_endpoint":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
91642025-02-28T12:10:10.704ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
91652025-02-28T12:10:10.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
91662025-02-28T12:10:10.706ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91672025-02-28T12:10:10.709ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
91682025-02-28T12:10:10.709ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
91692025-02-28T12:10:10.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
91702025-02-28T12:10:10.712ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91712025-02-28T12:10:10.714ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
91722025-02-28T12:10:10.714ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
91732025-02-28T12:10:10.715ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
91742025-02-28T12:10:10.724ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
91752025-02-28T12:10:10.802ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
91762025-02-28T12:10:10.802ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
91772025-02-28T12:10:10.802ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bcd552a-a639-47fe-bb3d-20d6181bb5c8", "content-length": "372", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} })
91782025-02-28T12:10:10.803ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
91792025-02-28T12:10:10.803ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91802025-02-28T12:10:10.847ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
91812025-02-28T12:10:10.847ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
91822025-02-28T12:10:10.847ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "38bde5bc-cfcb-4743-8f0a-7c5fd7f9a8d4", "content-length": "26", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} })
91832025-02-28T12:10:10.847ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
91842025-02-28T12:10:10.849ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
91852025-02-28T12:10:10.882ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
91862025-02-28T12:10:10.882ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
91872025-02-28T12:10:10.883ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "1bd01bf4-78ba-4d4b-b003-8992a37724c2", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} })
91882025-02-28T12:10:10.883ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
91892025-02-28T12:10:10.883ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91902025-02-28T12:10:10.947ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 493.634016ms iteration = 23 status = {}
91912025-02-28T12:10:10.958ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
91922025-02-28T12:10:10.985ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
91932025-02-28T12:10:10.985ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b428264-a60b-4951-ad8b-46bf7121f92a", "content-length": "1", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} })
91942025-02-28T12:10:10.986ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
91952025-02-28T12:10:10.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
91962025-02-28T12:10:11.063ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
91972025-02-28T12:10:11.064ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "b7bf6192-3e7e-4f54-a7d5-28bd72230b6b", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
91982025-02-28T12:10:11.064ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
91992025-02-28T12:10:11.065ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "75b9fd08-3233-49c5-921c-24ef8e17142e", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
92002025-02-28T12:10:11.065ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
92012025-02-28T12:10:11.067ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
92022025-02-28T12:10:11.067ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 613.817348ms iteration = 14 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
92032025-02-28T12:10:11.072ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
92042025-02-28T12:10:11.072ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "eff01925-b70d-4a8c-8033-dfab12b5d309", "content-length": "691", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} })
92052025-02-28T12:10:11.072ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
92062025-02-28T12:10:11.072ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
92072025-02-28T12:10:11.072ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
92082025-02-28T12:10:11.074ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 616.511613ms iteration = 23 status = {}
92092025-02-28T12:10:11.074ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
92102025-02-28T12:10:11.074ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
92112025-02-28T12:10:11.075ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 621.637677ms iteration = 23 status = null
92122025-02-28T12:10:11.076ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
92132025-02-28T12:10:11.077ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "1776a978-5687-43f5-be63-e744f7abb7f8", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
92142025-02-28T12:10:11.077ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
92152025-02-28T12:10:11.077ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
92162025-02-28T12:10:11.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
92172025-02-28T12:10:11.079ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "55a5dc40-f33e-4fc6-9142-b527db56474b", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
92182025-02-28T12:10:11.079ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
92192025-02-28T12:10:11.081ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "54ee61df-fa3c-4fc3-acb6-b996766fb50e", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
92202025-02-28T12:10:11.081ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
92212025-02-28T12:10:11.081ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
92222025-02-28T12:10:11.081ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
92232025-02-28T12:10:11.081ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "f87ebf4b-bbc4-4377-820d-bfd1a2055ec2", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} })
92242025-02-28T12:10:11.083ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
92252025-02-28T12:10:11.083ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 629.923676ms iteration = 14 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
92262025-02-28T12:10:11.083ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 626.163546ms iteration = 23 status = {}
92272025-02-28T12:10:11.129ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92282025-02-28T12:10:11.130ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92292025-02-28T12:10:11.132ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
92302025-02-28T12:10:11.132ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92312025-02-28T12:10:11.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92322025-02-28T12:10:11.141ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
92332025-02-28T12:10:11.142ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "760d4dea-6564-4242-8de3-c0ed6bef6838", "content-length": "134", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} })
92342025-02-28T12:10:11.142ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
92352025-02-28T12:10:11.142ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
92362025-02-28T12:10:11.142ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
92372025-02-28T12:10:11.142ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
92382025-02-28T12:10:11.142ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
92392025-02-28T12:10:11.142ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
92402025-02-28T12:10:11.142ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
92412025-02-28T12:10:11.142ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
92422025-02-28T12:10:11.142ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
92432025-02-28T12:10:11.142ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
92442025-02-28T12:10:11.142ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
92452025-02-28T12:10:11.142ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92462025-02-28T12:10:11.143ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92472025-02-28T12:10:11.145ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
92482025-02-28T12:10:11.145ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "f106c4db-cc95-4362-ba3f-c4b19aa76ab5", "content-length": "2", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} })
92492025-02-28T12:10:11.145ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
92502025-02-28T12:10:11.145ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "36a85198-e970-481e-a43a-109ed2a2616c", "content-length": "51", "date": "Fri, 28 Feb 2025 12:10:10 GMT"} })
92512025-02-28T12:10:11.145ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
92522025-02-28T12:10:11.145ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
92532025-02-28T12:10:11.290ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "71b730af-682b-468d-8ada-a0c8dc0b166f", "date": "Fri, 28 Feb 2025 12:10:11 GMT"} })
92542025-02-28T12:10:11.290ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
92552025-02-28T12:10:11.290ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
92562025-02-28T12:10:11.294ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1f6f56c-a1b8-4556-b44b-d56c423b1fde", "content-length": "682", "date": "Fri, 28 Feb 2025 12:10:11 GMT"} })
92572025-02-28T12:10:11.294ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92582025-02-28T12:10:11.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92592025-02-28T12:10:11.297ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92602025-02-28T12:10:11.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92612025-02-28T12:10:11.301ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92622025-02-28T12:10:11.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92632025-02-28T12:10:11.304ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
92642025-02-28T12:10:11.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
92652025-02-28T12:10:11.311ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 853.594276ms iteration = 23 status = {}
92662025-02-28T12:10:11.344ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
92672025-02-28T12:10:11.344ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::a]:12221
92682025-02-28T12:10:11.375ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
92692025-02-28T12:10:11.375ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::a]:60733
92702025-02-28T12:10:11.375ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:60733 req_id = cacf9cb2-1ce3-442c-b1b3-0d74243f9ebc roles = RoleSet { roles: {} } uri = /metrics/producers
92712025-02-28T12:10:11.375ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:60733 req_id = cacf9cb2-1ce3-442c-b1b3-0d74243f9ebc resource = Database result = Ok(()) uri = /metrics/producers
92722025-02-28T12:10:11.427ZINFOnexus: assigned collector to new producer collector_id = a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79 file = nexus/src/app/oximeter.rs:76 producer_id = c1b34bc5-1c2f-4d3c-808c-b7a441aac295
92732025-02-28T12:10:11.427ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 52261 local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:60733 req_id = cacf9cb2-1ce3-442c-b1b3-0d74243f9ebc response_code = 201 uri = /metrics/producers
92742025-02-28T12:10:11.427ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "cacf9cb2-1ce3-442c-b1b3-0d74243f9ebc", "content-length": "41", "date": "Fri, 28 Feb 2025 12:10:11 GMT"} })
92752025-02-28T12:10:11.428ZDEBGnexus (producer-registration-task): registered with nexus successfully
92762025-02-28T12:10:11.428ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
92772025-02-28T12:10:14.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 448 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = e291d384-b072-4620-bf84-1aca923056c9 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
92782025-02-28T12:10:20.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 68 reason = Timeout
92792025-02-28T12:10:20.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92802025-02-28T12:10:20.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92812025-02-28T12:10:20.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92822025-02-28T12:10:20.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 68 reason = Timeout
92832025-02-28T12:10:20.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
92842025-02-28T12:10:20.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92852025-02-28T12:10:20.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
92862025-02-28T12:10:20.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92872025-02-28T12:10:20.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92882025-02-28T12:10:20.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92892025-02-28T12:10:20.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92902025-02-28T12:10:20.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92912025-02-28T12:10:20.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
92922025-02-28T12:10:20.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92932025-02-28T12:10:20.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92942025-02-28T12:10:20.476ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
92952025-02-28T12:10:20.476ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.903925ms iteration = 68 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
92962025-02-28T12:10:20.490ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
92972025-02-28T12:10:20.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92982025-02-28T12:10:20.492ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.979014ms iteration = 68 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
92992025-02-28T12:10:24.253ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::d]:60730
93002025-02-28T12:10:24.253ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60730 req_id = 692ecaf3-9455-4c21-b411-5a04a089766d roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
93012025-02-28T12:10:24.254ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60730 req_id = 692ecaf3-9455-4c21-b411-5a04a089766d resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
93022025-02-28T12:10:24.257ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3438 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60730 req_id = 692ecaf3-9455-4c21-b411-5a04a089766d response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
93032025-02-28T12:10:24.257ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60730 req_id = f4e716d5-e231-4470-b09c-2bbd039dcf02 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
93042025-02-28T12:10:24.257ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60730 req_id = f4e716d5-e231-4470-b09c-2bbd039dcf02 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
93052025-02-28T12:10:24.259ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2268 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60730 req_id = f4e716d5-e231-4470-b09c-2bbd039dcf02 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
93062025-02-28T12:10:24.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 297 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = ebdf0978-22cc-4788-95c3-93c5f32ef858 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
93072025-02-28T12:10:30.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 69 reason = Timeout
93082025-02-28T12:10:30.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93092025-02-28T12:10:30.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93102025-02-28T12:10:30.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 69 reason = Timeout
93112025-02-28T12:10:30.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
93122025-02-28T12:10:30.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93132025-02-28T12:10:30.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
93142025-02-28T12:10:30.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
93152025-02-28T12:10:30.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
93162025-02-28T12:10:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
93172025-02-28T12:10:30.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93182025-02-28T12:10:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93192025-02-28T12:10:30.461ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.637848ms iteration = 69 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
93202025-02-28T12:10:30.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93212025-02-28T12:10:30.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93222025-02-28T12:10:30.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93232025-02-28T12:10:30.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
93242025-02-28T12:10:30.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93252025-02-28T12:10:30.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93262025-02-28T12:10:30.479ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
93272025-02-28T12:10:30.479ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.066427ms iteration = 69 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
93282025-02-28T12:10:34.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 418 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 2fd7e488-403b-4b05-b277-65f7f6aedb37 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
93292025-02-28T12:10:40.452ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 24 reason = Timeout
93302025-02-28T12:10:40.452ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
93312025-02-28T12:10:40.452ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 24 reason = Timeout
93322025-02-28T12:10:40.452ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
93332025-02-28T12:10:40.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
93342025-02-28T12:10:40.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
93352025-02-28T12:10:40.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 70 reason = Timeout
93362025-02-28T12:10:40.455ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 24 reason = Timeout
93372025-02-28T12:10:40.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93382025-02-28T12:10:40.455ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1475, tv_nsec: 92695850 } })
93392025-02-28T12:10:40.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93402025-02-28T12:10:40.455ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
93412025-02-28T12:10:40.456ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1475, tv_nsec: 93933790 } })
93422025-02-28T12:10:40.457ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
93432025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93442025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 70 reason = Timeout
93452025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
93462025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 24 reason = Timeout
93472025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
93482025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 24 reason = Timeout
93492025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
93502025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
93512025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 24 reason = Timeout
93522025-02-28T12:10:40.457ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
93532025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 24 reason = Timeout
93542025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
93552025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
93562025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 24 reason = Timeout
93572025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
93582025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 26 reason = Timeout
93592025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
93602025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 24 reason = Timeout
93612025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
93622025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 24 reason = Timeout
93632025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
93642025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 24 reason = Timeout
93652025-02-28T12:10:40.457ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
93662025-02-28T12:10:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
93672025-02-28T12:10:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
93682025-02-28T12:10:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
93692025-02-28T12:10:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
93702025-02-28T12:10:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
93712025-02-28T12:10:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
93722025-02-28T12:10:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
93732025-02-28T12:10:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
93742025-02-28T12:10:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
93752025-02-28T12:10:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
93762025-02-28T12:10:40.458ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.196657ms iteration = 24 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
93772025-02-28T12:10:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 24 reason = Timeout
93782025-02-28T12:10:40.458ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 24 reason = Timeout
93792025-02-28T12:10:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 24 reason = Timeout
93802025-02-28T12:10:40.458ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
93812025-02-28T12:10:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
93822025-02-28T12:10:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
93832025-02-28T12:10:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 24 reason = Timeout
93842025-02-28T12:10:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
93852025-02-28T12:10:40.459ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 24 reason = Timeout
93862025-02-28T12:10:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
93872025-02-28T12:10:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
93882025-02-28T12:10:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
93892025-02-28T12:10:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
93902025-02-28T12:10:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
93912025-02-28T12:10:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
93922025-02-28T12:10:40.460ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93932025-02-28T12:10:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
93942025-02-28T12:10:40.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93952025-02-28T12:10:40.460ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93962025-02-28T12:10:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93972025-02-28T12:10:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
93982025-02-28T12:10:40.462ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
93992025-02-28T12:10:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
94002025-02-28T12:10:40.463ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94012025-02-28T12:10:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
94022025-02-28T12:10:40.463ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
94032025-02-28T12:10:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
94042025-02-28T12:10:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
94052025-02-28T12:10:40.464ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
94062025-02-28T12:10:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
94072025-02-28T12:10:40.465ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94082025-02-28T12:10:40.466ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94092025-02-28T12:10:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
94102025-02-28T12:10:40.467ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
94112025-02-28T12:10:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
94122025-02-28T12:10:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
94132025-02-28T12:10:40.468ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
94142025-02-28T12:10:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
94152025-02-28T12:10:40.470ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
94162025-02-28T12:10:40.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
94172025-02-28T12:10:40.471ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
94182025-02-28T12:10:40.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
94192025-02-28T12:10:40.472ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
94202025-02-28T12:10:40.472ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
94212025-02-28T12:10:40.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
94222025-02-28T12:10:40.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
94232025-02-28T12:10:40.472ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
94242025-02-28T12:10:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
94252025-02-28T12:10:40.474ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1475, tv_nsec: 111608606 } })
94262025-02-28T12:10:40.474ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
94272025-02-28T12:10:40.474ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
94282025-02-28T12:10:40.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
94292025-02-28T12:10:40.474ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
94302025-02-28T12:10:40.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
94312025-02-28T12:10:40.476ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94322025-02-28T12:10:40.479ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
94332025-02-28T12:10:40.480ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
94342025-02-28T12:10:40.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
94352025-02-28T12:10:40.506ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
94362025-02-28T12:10:40.506ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
94372025-02-28T12:10:40.506ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
94382025-02-28T12:10:40.506ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
94392025-02-28T12:10:40.507ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94402025-02-28T12:10:40.508ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
94412025-02-28T12:10:40.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
94422025-02-28T12:10:40.509ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
94432025-02-28T12:10:40.509ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
94442025-02-28T12:10:40.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
94452025-02-28T12:10:40.510ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 53.023351ms iteration = 70 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
94462025-02-28T12:10:40.515ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94472025-02-28T12:10:40.516ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 57.440767ms iteration = 24 status = {"errors":[],"garbage_collect_requested":[]}
94482025-02-28T12:10:40.517ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 58.137544ms iteration = 24 status = {"errors":[],"requests_created_ok":[]}
94492025-02-28T12:10:40.517ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
94502025-02-28T12:10:40.517ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
94512025-02-28T12:10:40.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
94522025-02-28T12:10:40.521ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
94532025-02-28T12:10:40.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
94542025-02-28T12:10:40.523ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
94552025-02-28T12:10:40.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
94562025-02-28T12:10:40.524ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94572025-02-28T12:10:40.525ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
94582025-02-28T12:10:40.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
94592025-02-28T12:10:40.526ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 67.744425ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
94602025-02-28T12:10:40.527ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
94612025-02-28T12:10:40.527ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
94622025-02-28T12:10:40.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
94632025-02-28T12:10:40.527ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94642025-02-28T12:10:40.528ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
94652025-02-28T12:10:40.528ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 69.869257ms iteration = 24 status = {"errors":[],"finish_invoked_ok":[]}
94662025-02-28T12:10:40.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
94672025-02-28T12:10:40.529ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
94682025-02-28T12:10:40.529ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 71.458495ms iteration = 24 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
94692025-02-28T12:10:40.530ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
94702025-02-28T12:10:40.530ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
94712025-02-28T12:10:40.530ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
94722025-02-28T12:10:40.533ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 74.744755ms iteration = 24 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
94732025-02-28T12:10:40.534ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
94742025-02-28T12:10:40.534ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 76.142659ms iteration = 24 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
94752025-02-28T12:10:40.551ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
94762025-02-28T12:10:40.551ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
94772025-02-28T12:10:40.551ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
94782025-02-28T12:10:40.552ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
94792025-02-28T12:10:40.555ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 97.988646ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
94802025-02-28T12:10:40.580ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
94812025-02-28T12:10:40.580ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
94822025-02-28T12:10:40.580ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "861c0bdf-3c1e-4c87-a523-e44ac4ace382", "content-length": "26", "date": "Fri, 28 Feb 2025 12:10:40 GMT"} })
94832025-02-28T12:10:40.580ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
94842025-02-28T12:10:40.582ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
94852025-02-28T12:10:40.647ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
94862025-02-28T12:10:40.647ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
94872025-02-28T12:10:40.647ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a007427d-a671-4d24-a86d-631312b7ec57", "content-length": "26", "date": "Fri, 28 Feb 2025 12:10:40 GMT"} })
94882025-02-28T12:10:40.647ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
94892025-02-28T12:10:40.649ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
94902025-02-28T12:10:40.683ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
94912025-02-28T12:10:40.683ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
94922025-02-28T12:10:40.684ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6aa2cb45-bfdb-4602-a687-a948acf49703", "content-length": "26", "date": "Fri, 28 Feb 2025 12:10:40 GMT"} })
94932025-02-28T12:10:40.684ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
94942025-02-28T12:10:40.685ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
94952025-02-28T12:10:40.689ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
94962025-02-28T12:10:40.689ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
94972025-02-28T12:10:40.690ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "8316f6ed-5cb7-4747-b971-1fd7adff7add", "content-length": "372", "date": "Fri, 28 Feb 2025 12:10:40 GMT"} })
94982025-02-28T12:10:40.690ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
94992025-02-28T12:10:40.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95002025-02-28T12:10:40.693ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
95012025-02-28T12:10:40.693ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9c5dbbe-2cd0-45f5-907c-30a6504f119e", "content-length": "1", "date": "Fri, 28 Feb 2025 12:10:40 GMT"} })
95022025-02-28T12:10:40.695ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
95032025-02-28T12:10:40.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
95042025-02-28T12:10:40.697ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 244.867461ms iteration = 24 status = null
95052025-02-28T12:10:40.712ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
95062025-02-28T12:10:40.712ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 256.788699ms iteration = 70 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
95072025-02-28T12:10:40.726ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95082025-02-28T12:10:40.730ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
95092025-02-28T12:10:40.730ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
95102025-02-28T12:10:40.730ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
95112025-02-28T12:10:40.733ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 275.445889ms iteration = 24 status = 0
95122025-02-28T12:10:40.736ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
95132025-02-28T12:10:40.736ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
95142025-02-28T12:10:40.736ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 278.204476ms iteration = 24 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
95152025-02-28T12:10:40.742ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 289.185951ms iteration = 24 status = {}
95162025-02-28T12:10:40.762ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
95172025-02-28T12:10:40.762ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
95182025-02-28T12:10:40.762ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "b2317dc5-1d81-411a-9247-8827af986dd6", "date": "Fri, 28 Feb 2025 12:10:40 GMT"} })
95192025-02-28T12:10:40.762ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95202025-02-28T12:10:40.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95212025-02-28T12:10:40.770ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95222025-02-28T12:10:40.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95232025-02-28T12:10:40.774ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95242025-02-28T12:10:40.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95252025-02-28T12:10:40.779ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
95262025-02-28T12:10:40.779ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
95272025-02-28T12:10:40.779ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "19efcb73-4516-45d7-9f3f-da16da2b7538", "date": "Fri, 28 Feb 2025 12:10:40 GMT"} })
95282025-02-28T12:10:40.781ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 323.811216ms iteration = 24 status = {}
95292025-02-28T12:10:40.833ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95302025-02-28T12:10:40.834ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95312025-02-28T12:10:40.836ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
95322025-02-28T12:10:40.836ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95332025-02-28T12:10:40.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95342025-02-28T12:10:40.844ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
95352025-02-28T12:10:40.844ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
95362025-02-28T12:10:40.846ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95372025-02-28T12:10:40.849ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
95382025-02-28T12:10:40.849ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
95392025-02-28T12:10:40.849ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
95402025-02-28T12:10:40.850ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
95412025-02-28T12:10:40.850ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0c87aca-b514-446b-81ce-98796fa67696", "content-length": "134", "date": "Fri, 28 Feb 2025 12:10:40 GMT"} })
95422025-02-28T12:10:40.850ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
95432025-02-28T12:10:40.850ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
95442025-02-28T12:10:40.850ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
95452025-02-28T12:10:40.850ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
95462025-02-28T12:10:40.850ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
95472025-02-28T12:10:40.850ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
95482025-02-28T12:10:40.850ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
95492025-02-28T12:10:40.850ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
95502025-02-28T12:10:40.851ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
95512025-02-28T12:10:40.851ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
95522025-02-28T12:10:40.851ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
95532025-02-28T12:10:40.851ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95542025-02-28T12:10:40.851ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95552025-02-28T12:10:40.853ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
95562025-02-28T12:10:40.853ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbcf8d2d-57ca-4a1a-967b-284bd15f94b4", "content-length": "2", "date": "Fri, 28 Feb 2025 12:10:40 GMT"} })
95572025-02-28T12:10:40.853ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
95582025-02-28T12:10:40.853ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "af85702d-04d3-406c-90d6-b4df68080838", "content-length": "51", "date": "Fri, 28 Feb 2025 12:10:40 GMT"} })
95592025-02-28T12:10:40.853ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
95602025-02-28T12:10:40.853ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
95612025-02-28T12:10:40.859ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 401.277417ms iteration = 26 status = {"physical_disks_added":0}
95622025-02-28T12:10:40.891ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95632025-02-28T12:10:40.895ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
95642025-02-28T12:10:40.895ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
95652025-02-28T12:10:40.895ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
95662025-02-28T12:10:40.983ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "5751f04b-79bf-4683-a139-907ed82744c5", "date": "Fri, 28 Feb 2025 12:10:40 GMT"} })
95672025-02-28T12:10:40.983ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
95682025-02-28T12:10:40.983ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
95692025-02-28T12:10:40.984ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ac5094a-7c11-4d46-b828-1fd424f8a6fd", "content-length": "682", "date": "Fri, 28 Feb 2025 12:10:40 GMT"} })
95702025-02-28T12:10:40.984ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95712025-02-28T12:10:40.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95722025-02-28T12:10:40.988ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95732025-02-28T12:10:40.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95742025-02-28T12:10:40.993ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95752025-02-28T12:10:40.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95762025-02-28T12:10:40.995ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95772025-02-28T12:10:40.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95782025-02-28T12:10:41.006ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 548.572802ms iteration = 24 status = {}
95792025-02-28T12:10:41.009ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
95802025-02-28T12:10:41.143ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7cbab45-bab1-493b-9853-cfa5819951f7", "content-length": "691", "date": "Fri, 28 Feb 2025 12:10:41 GMT"} })
95812025-02-28T12:10:41.143ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
95822025-02-28T12:10:41.143ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
95832025-02-28T12:10:41.143ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
95842025-02-28T12:10:41.146ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 688.257086ms iteration = 24 status = {}
95852025-02-28T12:10:44.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 281 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 6ae4bdb7-b017-437d-8baf-d217f522d37e response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
95862025-02-28T12:10:50.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 71 reason = Timeout
95872025-02-28T12:10:50.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
95882025-02-28T12:10:50.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
95892025-02-28T12:10:50.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 71 reason = Timeout
95902025-02-28T12:10:50.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
95912025-02-28T12:10:50.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95922025-02-28T12:10:50.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
95932025-02-28T12:10:50.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
95942025-02-28T12:10:50.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
95952025-02-28T12:10:50.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
95962025-02-28T12:10:50.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
95972025-02-28T12:10:50.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
95982025-02-28T12:10:50.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95992025-02-28T12:10:50.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
96002025-02-28T12:10:50.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96012025-02-28T12:10:50.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96022025-02-28T12:10:50.484ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
96032025-02-28T12:10:50.484ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 28.284593ms iteration = 71 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
96042025-02-28T12:10:50.499ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
96052025-02-28T12:10:50.499ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
96062025-02-28T12:10:50.532ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 74.53472ms iteration = 71 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
96072025-02-28T12:10:54.252ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60730 req_id = c79897f3-f357-424c-9ef6-9cf42c1b3d87 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
96082025-02-28T12:10:54.253ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60730 req_id = c79897f3-f357-424c-9ef6-9cf42c1b3d87 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
96092025-02-28T12:10:54.256ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3326 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60730 req_id = c79897f3-f357-424c-9ef6-9cf42c1b3d87 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
96102025-02-28T12:10:54.256ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60730 req_id = 5bb7cf38-38e0-4067-8fa6-31859a296de1 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
96112025-02-28T12:10:54.256ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60730 req_id = 5bb7cf38-38e0-4067-8fa6-31859a296de1 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
96122025-02-28T12:10:54.259ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2667 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60730 req_id = 5bb7cf38-38e0-4067-8fa6-31859a296de1 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
96132025-02-28T12:10:54.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 233 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 7ceb5662-d0f1-4e90-964a-a92ee5587e25 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
96142025-02-28T12:11:00.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 72 reason = Timeout
96152025-02-28T12:11:00.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96162025-02-28T12:11:00.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96172025-02-28T12:11:00.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96182025-02-28T12:11:00.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 72 reason = Timeout
96192025-02-28T12:11:00.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
96202025-02-28T12:11:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
96212025-02-28T12:11:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
96222025-02-28T12:11:00.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96232025-02-28T12:11:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96242025-02-28T12:11:00.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96252025-02-28T12:11:00.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96262025-02-28T12:11:00.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96272025-02-28T12:11:00.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
96282025-02-28T12:11:00.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96292025-02-28T12:11:00.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96302025-02-28T12:11:00.481ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
96312025-02-28T12:11:00.481ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.479765ms iteration = 72 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
96322025-02-28T12:11:00.491ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
96332025-02-28T12:11:00.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
96342025-02-28T12:11:00.522ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.984619ms iteration = 72 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
96352025-02-28T12:11:04.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 300 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = efebcc70-8f7c-46d9-82c3-b023e144c360 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
96362025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 13 reason = Timeout
96372025-02-28T12:11:10.453ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-28 12:01:10.453104115 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
96382025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
96392025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 13 reason = Timeout
96402025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
96412025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 25 reason = Timeout
96422025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
96432025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
96442025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 15 reason = Timeout
96452025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 13 reason = Timeout
96462025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 13 reason = Timeout
96472025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 13 reason = Timeout
96482025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
96492025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
96502025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 13 reason = Timeout
96512025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
96522025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 25 reason = Timeout
96532025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 15 reason = Timeout
96542025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
96552025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
96562025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
96572025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
96582025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
96592025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
96602025-02-28T12:11:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
96612025-02-28T12:11:10.453ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5ebff2d9-48b6-4fe6-b852-684bb14b4955.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bddfb0f9-1d7b-45e4-b3a5-35aa6792956b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1505, tv_nsec: 91080031 } })
96622025-02-28T12:11:10.453ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5cefabf1-6831-46ff-98eb-1a3b5abad8b9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1505, tv_nsec: 91138882 } })
96632025-02-28T12:11:10.454ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
96642025-02-28T12:11:10.454ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.163879ms iteration = 13 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
96652025-02-28T12:11:10.454ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
96662025-02-28T12:11:10.454ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.25209ms iteration = 13 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
96672025-02-28T12:11:10.454ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
96682025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
96692025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 73 reason = Timeout
96702025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 14 reason = Timeout
96712025-02-28T12:11:10.455ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
96722025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96732025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
96742025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 11.78\u{b5}s iteration = 14 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
96752025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 6 reason = Timeout
96762025-02-28T12:11:10.455ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-28 12:01:10.453104115 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
96772025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
96782025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 25 reason = Timeout
96792025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 2.347197ms iteration = 13 status = {"expiration":"2025-02-28T12:01:10.453104115Z","pruned":[]}
96802025-02-28T12:11:10.455ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1505, tv_nsec: 92963397 } })
96812025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96822025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
96832025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
96842025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
96852025-02-28T12:11:10.455ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
96862025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
96872025-02-28T12:11:10.455ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
96882025-02-28T12:11:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
96892025-02-28T12:11:10.456ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
96902025-02-28T12:11:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
96912025-02-28T12:11:10.456ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
96922025-02-28T12:11:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
96932025-02-28T12:11:10.456ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96942025-02-28T12:11:10.457ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1505, tv_nsec: 94701728 } })
96952025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 13 reason = Timeout
96962025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 25 reason = Timeout
96972025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
96982025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
96992025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 25 reason = Timeout
97002025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 25 reason = Timeout
97012025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
97022025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
97032025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 25 reason = Timeout
97042025-02-28T12:11:10.457ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
97052025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
97062025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 13 reason = Timeout
97072025-02-28T12:11:10.457ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
97082025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 27 reason = Timeout
97092025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 25 reason = Timeout
97102025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
97112025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
97122025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
97132025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 25 reason = Timeout
97142025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
97152025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 25 reason = Timeout
97162025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
97172025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
97182025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 25 reason = Timeout
97192025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
97202025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97212025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 73 reason = Timeout
97222025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
97232025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
97242025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
97252025-02-28T12:11:10.457ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
97262025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
97272025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
97282025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
97292025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
97302025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
97312025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
97322025-02-28T12:11:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
97332025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
97342025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
97352025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
97362025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97372025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 25 reason = Timeout
97382025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
97392025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 25 reason = Timeout
97402025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
97412025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 25 reason = Timeout
97422025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
97432025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 25 reason = Timeout
97442025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
97452025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 13 reason = Timeout
97462025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
97472025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 25 reason = Timeout
97482025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
97492025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 13 reason = Timeout
97502025-02-28T12:11:10.458ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
97512025-02-28T12:11:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
97522025-02-28T12:11:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
97532025-02-28T12:11:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
97542025-02-28T12:11:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
97552025-02-28T12:11:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
97562025-02-28T12:11:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
97572025-02-28T12:11:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
97582025-02-28T12:11:10.459ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
97592025-02-28T12:11:10.459ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
97602025-02-28T12:11:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
97612025-02-28T12:11:10.460ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.624612ms iteration = 25 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
97622025-02-28T12:11:10.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
97632025-02-28T12:11:10.461ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97642025-02-28T12:11:10.461ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97652025-02-28T12:11:10.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
97662025-02-28T12:11:10.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97672025-02-28T12:11:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97682025-02-28T12:11:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
97692025-02-28T12:11:10.464ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
97702025-02-28T12:11:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
97712025-02-28T12:11:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
97722025-02-28T12:11:10.465ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97732025-02-28T12:11:10.466ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97742025-02-28T12:11:10.468ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
97752025-02-28T12:11:10.468ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
97762025-02-28T12:11:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
97772025-02-28T12:11:10.468ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
97782025-02-28T12:11:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
97792025-02-28T12:11:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
97802025-02-28T12:11:10.468ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97812025-02-28T12:11:10.469ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97822025-02-28T12:11:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
97832025-02-28T12:11:10.470ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
97842025-02-28T12:11:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
97852025-02-28T12:11:10.471ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97862025-02-28T12:11:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
97872025-02-28T12:11:10.472ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
97882025-02-28T12:11:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
97892025-02-28T12:11:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
97902025-02-28T12:11:10.473ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
97912025-02-28T12:11:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
97922025-02-28T12:11:10.474ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
97932025-02-28T12:11:10.474ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97942025-02-28T12:11:10.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
97952025-02-28T12:11:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
97962025-02-28T12:11:10.476ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
97972025-02-28T12:11:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
97982025-02-28T12:11:10.485ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 27.002661ms iteration = 13 status = {"errors":[],"found_port_ok":[]}
97992025-02-28T12:11:10.486ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 27.491583ms iteration = 25 status = {"errors":[],"garbage_collect_requested":[]}
98002025-02-28T12:11:10.488ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98012025-02-28T12:11:10.488ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
98022025-02-28T12:11:10.488ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
98032025-02-28T12:11:10.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
98042025-02-28T12:11:10.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
98052025-02-28T12:11:10.491ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
98062025-02-28T12:11:10.491ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
98072025-02-28T12:11:10.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
98082025-02-28T12:11:10.499ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1505, tv_nsec: 137024780 } })
98092025-02-28T12:11:10.500ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
98102025-02-28T12:11:10.515ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
98112025-02-28T12:11:10.515ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
98122025-02-28T12:11:10.515ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
98132025-02-28T12:11:10.515ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
98142025-02-28T12:11:10.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
98152025-02-28T12:11:10.519ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
98162025-02-28T12:11:10.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
98172025-02-28T12:11:10.519ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98182025-02-28T12:11:10.519ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98192025-02-28T12:11:10.521ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98202025-02-28T12:11:10.521ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
98212025-02-28T12:11:10.521ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
98222025-02-28T12:11:10.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
98232025-02-28T12:11:10.521ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98242025-02-28T12:11:10.523ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
98252025-02-28T12:11:10.523ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
98262025-02-28T12:11:10.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
98272025-02-28T12:11:10.524ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
98282025-02-28T12:11:10.524ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
98292025-02-28T12:11:10.524ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
98302025-02-28T12:11:10.524ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
98312025-02-28T12:11:10.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
98322025-02-28T12:11:10.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
98332025-02-28T12:11:10.525ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98342025-02-28T12:11:10.525ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
98352025-02-28T12:11:10.525ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
98362025-02-28T12:11:10.526ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
98372025-02-28T12:11:10.526ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
98382025-02-28T12:11:10.526ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
98392025-02-28T12:11:10.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
98402025-02-28T12:11:10.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
98412025-02-28T12:11:10.526ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98422025-02-28T12:11:10.527ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
98432025-02-28T12:11:10.527ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
98442025-02-28T12:11:10.527ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
98452025-02-28T12:11:10.527ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
98462025-02-28T12:11:10.527ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 69.119309ms iteration = 13 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
98472025-02-28T12:11:10.527ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98482025-02-28T12:11:10.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
98492025-02-28T12:11:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
98502025-02-28T12:11:10.528ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
98512025-02-28T12:11:10.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
98522025-02-28T12:11:10.530ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
98532025-02-28T12:11:10.530ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
98542025-02-28T12:11:10.530ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
98552025-02-28T12:11:10.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
98562025-02-28T12:11:10.531ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
98572025-02-28T12:11:10.531ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
98582025-02-28T12:11:10.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
98592025-02-28T12:11:10.532ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98602025-02-28T12:11:10.536ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
98612025-02-28T12:11:10.536ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
98622025-02-28T12:11:10.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
98632025-02-28T12:11:10.537ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 78.172618ms iteration = 25 status = {"errors":[],"requests_created_ok":[]}
98642025-02-28T12:11:10.537ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
98652025-02-28T12:11:10.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
98662025-02-28T12:11:10.537ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 78.525636ms iteration = 25 status = {"errors":[],"finish_invoked_ok":[]}
98672025-02-28T12:11:10.537ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
98682025-02-28T12:11:10.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
98692025-02-28T12:11:10.537ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
98702025-02-28T12:11:10.537ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
98712025-02-28T12:11:10.537ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
98722025-02-28T12:11:10.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
98732025-02-28T12:11:10.539ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 81.999ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
98742025-02-28T12:11:10.539ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 84.189133ms iteration = 6 status = {"nsuccess":5}
98752025-02-28T12:11:10.541ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
98762025-02-28T12:11:10.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
98772025-02-28T12:11:10.542ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
98782025-02-28T12:11:10.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
98792025-02-28T12:11:10.544ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98802025-02-28T12:11:10.546ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
98812025-02-28T12:11:10.546ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
98822025-02-28T12:11:10.546ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
98832025-02-28T12:11:10.548ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
98842025-02-28T12:11:10.548ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 90.629437ms iteration = 25 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
98852025-02-28T12:11:10.549ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
98862025-02-28T12:11:10.549ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 93.754293ms iteration = 73 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
98872025-02-28T12:11:10.551ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 93.803724ms iteration = 73 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
98882025-02-28T12:11:10.560ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
98892025-02-28T12:11:10.560ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
98902025-02-28T12:11:10.567ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
98912025-02-28T12:11:10.567ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
98922025-02-28T12:11:10.570ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
98932025-02-28T12:11:10.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
98942025-02-28T12:11:10.571ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98952025-02-28T12:11:10.572ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98962025-02-28T12:11:10.574ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
98972025-02-28T12:11:10.574ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
98982025-02-28T12:11:10.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
98992025-02-28T12:11:10.574ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
99002025-02-28T12:11:10.574ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
99012025-02-28T12:11:10.575ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
99022025-02-28T12:11:10.576ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 117.506107ms iteration = 25 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
99032025-02-28T12:11:10.576ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 119.488724ms iteration = 25 status = 0
99042025-02-28T12:11:10.577ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
99052025-02-28T12:11:10.577ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
99062025-02-28T12:11:10.577ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
99072025-02-28T12:11:10.577ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
99082025-02-28T12:11:10.579ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99092025-02-28T12:11:10.582ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
99102025-02-28T12:11:10.582ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 125.160591ms iteration = 25 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
99112025-02-28T12:11:10.584ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
99122025-02-28T12:11:10.584ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
99132025-02-28T12:11:10.584ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
99142025-02-28T12:11:10.585ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
99152025-02-28T12:11:10.585ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
99162025-02-28T12:11:10.585ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
99172025-02-28T12:11:10.585ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99182025-02-28T12:11:10.585ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
99192025-02-28T12:11:10.586ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99202025-02-28T12:11:10.587ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
99212025-02-28T12:11:10.587ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
99222025-02-28T12:11:10.587ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
99232025-02-28T12:11:10.587ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
99242025-02-28T12:11:10.587ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-02-28 11:59:51.915187 UTC dns_group = internal generation = 1
99252025-02-28T12:11:10.587ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 134.515157ms iteration = 13 status = {"generation":1}
99262025-02-28T12:11:10.587ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
99272025-02-28T12:11:10.588ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 130.051029ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
99282025-02-28T12:11:10.590ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
99292025-02-28T12:11:10.590ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
99302025-02-28T12:11:10.590ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
99312025-02-28T12:11:10.600ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
99322025-02-28T12:11:10.600ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
99332025-02-28T12:11:10.600ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 143.29777ms iteration = 13 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
99342025-02-28T12:11:10.613ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
99352025-02-28T12:11:10.613ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
99362025-02-28T12:11:10.613ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "432984f1-fd89-408a-bff7-03f21a5a5b4e", "content-length": "26", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} })
99372025-02-28T12:11:10.613ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
99382025-02-28T12:11:10.616ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
99392025-02-28T12:11:10.636ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-28T11:59:08.719399Z, time_modified: 2025-02-28T11:59:08.719399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:94
99402025-02-28T12:11:10.636ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 183.11728ms iteration = 13 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]}},"default_endpoint":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
99412025-02-28T12:11:10.637ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
99422025-02-28T12:11:10.637ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
99432025-02-28T12:11:10.637ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 179.566154ms iteration = 25 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
99442025-02-28T12:11:10.639ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
99452025-02-28T12:11:10.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
99462025-02-28T12:11:10.651ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
99472025-02-28T12:11:10.651ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 194.184567ms iteration = 13 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
99482025-02-28T12:11:10.663ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
99492025-02-28T12:11:10.663ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 0 version = 2
99502025-02-28T12:11:10.663ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
99512025-02-28T12:11:10.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
99522025-02-28T12:11:10.665ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99532025-02-28T12:11:10.669ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
99542025-02-28T12:11:10.669ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
99552025-02-28T12:11:10.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
99562025-02-28T12:11:10.673ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
99572025-02-28T12:11:10.677ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 1 version = 2
99582025-02-28T12:11:10.677ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
99592025-02-28T12:11:10.677ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-02-28 11:59:52.542241 UTC dns_group = external generation = 2
99602025-02-28T12:11:10.677ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 223.991637ms iteration = 13 status = {"generation":2}
99612025-02-28T12:11:10.711ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
99622025-02-28T12:11:10.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
99632025-02-28T12:11:10.713ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
99642025-02-28T12:11:10.713ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
99652025-02-28T12:11:10.713ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99662025-02-28T12:11:10.714ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5134bac-0790-4ba4-891e-410920a802b8", "content-length": "26", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} })
99672025-02-28T12:11:10.714ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
99682025-02-28T12:11:10.714ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
99692025-02-28T12:11:10.714ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
99702025-02-28T12:11:10.715ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8ee898e-139f-4b66-b890-3b4961684e06", "content-length": "372", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} })
99712025-02-28T12:11:10.715ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
99722025-02-28T12:11:10.715ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
99732025-02-28T12:11:10.716ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
99742025-02-28T12:11:10.716ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
99752025-02-28T12:11:10.717ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
99762025-02-28T12:11:10.717ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
99772025-02-28T12:11:10.718ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99782025-02-28T12:11:10.721ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
99792025-02-28T12:11:10.721ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
99802025-02-28T12:11:10.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
99812025-02-28T12:11:10.746ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {0c4c8e83-f0c6-4194-926d-035e56ca7357: {10.151.2.175: {aa7bd82a-6138-4a4e-a875-aa82b863a7d2}, 10.151.2.176: {aa7bd82a-6138-4a4e-a875-aa82b863a7d2}}} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
99822025-02-28T12:11:10.746ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
99832025-02-28T12:11:10.746ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "c7e1cf49-1cb8-445a-9fcd-0a1768e17280", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} })
99842025-02-28T12:11:10.746ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
99852025-02-28T12:11:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
99862025-02-28T12:11:10.748ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
99872025-02-28T12:11:10.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
99882025-02-28T12:11:10.758ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
99892025-02-28T12:11:10.770ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 312.671048ms iteration = 27 status = {"physical_disks_added":0}
99902025-02-28T12:11:10.770ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
99912025-02-28T12:11:10.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
99922025-02-28T12:11:10.776ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
99932025-02-28T12:11:10.776ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
99942025-02-28T12:11:10.777ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "6f45d84e-bd0d-4b1d-aaca-d86b19da9b42", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} })
99952025-02-28T12:11:10.780ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 322.693479ms iteration = 25 status = {}
99962025-02-28T12:11:10.786ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
99972025-02-28T12:11:10.786ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
99982025-02-28T12:11:10.787ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3213d154-daa0-477c-9c8e-f87d8e922e3f", "content-length": "26", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} })
99992025-02-28T12:11:10.787ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
100002025-02-28T12:11:10.789ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
100012025-02-28T12:11:10.902ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
100022025-02-28T12:11:10.902ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "42608062-dfce-4dbf-a940-dfc922358f19", "content-length": "2", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} })
100032025-02-28T12:11:10.904ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
100042025-02-28T12:11:10.905ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
100052025-02-28T12:11:10.949ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
100062025-02-28T12:11:10.953ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 500.086212ms iteration = 25 status = {}
100072025-02-28T12:11:10.955ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
100082025-02-28T12:11:10.955ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "7390b0d8-2237-438a-9066-1a43e967b21b", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
100092025-02-28T12:11:10.955ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
100102025-02-28T12:11:10.957ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "0f13d690-2893-4dc1-8a8b-a0d6f5eab154", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
100112025-02-28T12:11:10.957ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
100122025-02-28T12:11:10.959ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
100132025-02-28T12:11:10.959ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 505.906692ms iteration = 15 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
100142025-02-28T12:11:10.989ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
100152025-02-28T12:11:10.989ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "07fc4070-ddfb-4ce8-a1fe-9c8fcb913698", "content-length": "691", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} })
100162025-02-28T12:11:10.989ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
100172025-02-28T12:11:10.989ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
100182025-02-28T12:11:10.989ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [VirtualNetworkInterfaceHost { physical_host_ip: fd00:1122:3344:101::1, virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), vni: Vni(9653006) }] sled = sock
100192025-02-28T12:11:10.989ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/v2p
100202025-02-28T12:11:10.990ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 204, headers: {"x-request-id": "22772c89-0197-4efb-bb63-899d28ef7116", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} })
100212025-02-28T12:11:10.991ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 533.832997ms iteration = 25 status = {}
100222025-02-28T12:11:10.992ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 538.958981ms iteration = 25 status = null
100232025-02-28T12:11:11.033ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
100242025-02-28T12:11:11.033ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "4fbd4680-1d9e-47c4-b78d-69397d3c1d63", "date": "Fri, 28 Feb 2025 12:11:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
100252025-02-28T12:11:11.033ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
100262025-02-28T12:11:11.036ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "e216d367-5cc6-4f3e-bb7a-6f79d261c586", "date": "Fri, 28 Feb 2025 12:11:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
100272025-02-28T12:11:11.036ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
100282025-02-28T12:11:11.037ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "3b2e2ede-027d-4e55-af6c-c550770847d1", "date": "Fri, 28 Feb 2025 12:11:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
100292025-02-28T12:11:11.037ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
100302025-02-28T12:11:11.039ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
100312025-02-28T12:11:11.039ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 586.384506ms iteration = 15 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
100322025-02-28T12:11:11.043ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100332025-02-28T12:11:11.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100342025-02-28T12:11:11.046ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
100352025-02-28T12:11:11.046ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100362025-02-28T12:11:11.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100372025-02-28T12:11:11.057ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
100382025-02-28T12:11:11.057ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "de4e55f8-2c9f-4450-856b-a094c1a8df72", "content-length": "134", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} })
100392025-02-28T12:11:11.057ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
100402025-02-28T12:11:11.057ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
100412025-02-28T12:11:11.058ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
100422025-02-28T12:11:11.058ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
100432025-02-28T12:11:11.058ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
100442025-02-28T12:11:11.058ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
100452025-02-28T12:11:11.058ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
100462025-02-28T12:11:11.058ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
100472025-02-28T12:11:11.058ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
100482025-02-28T12:11:11.058ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
100492025-02-28T12:11:11.058ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
100502025-02-28T12:11:11.058ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100512025-02-28T12:11:11.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100522025-02-28T12:11:11.061ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
100532025-02-28T12:11:11.061ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb2d71fc-2642-40d1-976a-06571e20f02c", "content-length": "2", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} })
100542025-02-28T12:11:11.061ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
100552025-02-28T12:11:11.061ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fe4539e-1900-4bbd-b6fc-96e75df33373", "content-length": "51", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} })
100562025-02-28T12:11:11.061ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
100572025-02-28T12:11:11.061ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
100582025-02-28T12:11:11.163ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "73ecf132-eaf7-4ad1-81b7-a47dedea6e83", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} })
100592025-02-28T12:11:11.163ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
100602025-02-28T12:11:11.163ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
100612025-02-28T12:11:11.164ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "74f23807-f8e2-4d6a-90ef-4d24d8909a26", "content-length": "682", "date": "Fri, 28 Feb 2025 12:11:10 GMT"} })
100622025-02-28T12:11:11.164ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100632025-02-28T12:11:11.164ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100642025-02-28T12:11:11.168ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100652025-02-28T12:11:11.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100662025-02-28T12:11:11.172ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100672025-02-28T12:11:11.172ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100682025-02-28T12:11:11.174ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100692025-02-28T12:11:11.175ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100702025-02-28T12:11:11.184ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 726.592231ms iteration = 25 status = {}
100712025-02-28T12:11:14.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 262 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 7f5474c0-88a0-484b-9661-c5ceb70d6b1a response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
100722025-02-28T12:11:20.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 74 reason = Timeout
100732025-02-28T12:11:20.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100742025-02-28T12:11:20.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100752025-02-28T12:11:20.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 74 reason = Timeout
100762025-02-28T12:11:20.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
100772025-02-28T12:11:20.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100782025-02-28T12:11:20.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
100792025-02-28T12:11:20.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
100802025-02-28T12:11:20.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100812025-02-28T12:11:20.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100822025-02-28T12:11:20.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100832025-02-28T12:11:20.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100842025-02-28T12:11:20.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100852025-02-28T12:11:20.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
100862025-02-28T12:11:20.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100872025-02-28T12:11:20.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100882025-02-28T12:11:20.478ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
100892025-02-28T12:11:20.478ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.225231ms iteration = 74 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
100902025-02-28T12:11:20.489ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
100912025-02-28T12:11:20.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
100922025-02-28T12:11:20.490ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.1158ms iteration = 74 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
100932025-02-28T12:11:24.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 253 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = f3c6f540-c25c-44fd-8947-a36447decc9a response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
100942025-02-28T12:11:30.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 75 reason = Timeout
100952025-02-28T12:11:30.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100962025-02-28T12:11:30.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100972025-02-28T12:11:30.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 75 reason = Timeout
100982025-02-28T12:11:30.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
100992025-02-28T12:11:30.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
101002025-02-28T12:11:30.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101012025-02-28T12:11:30.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
101022025-02-28T12:11:30.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
101032025-02-28T12:11:30.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
101042025-02-28T12:11:30.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
101052025-02-28T12:11:30.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
101062025-02-28T12:11:30.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101072025-02-28T12:11:30.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
101082025-02-28T12:11:30.468ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
101092025-02-28T12:11:30.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
101102025-02-28T12:11:30.488ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
101112025-02-28T12:11:30.488ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 31.792378ms iteration = 75 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
101122025-02-28T12:11:30.498ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
101132025-02-28T12:11:30.499ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
101142025-02-28T12:11:30.501ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 43.753997ms iteration = 75 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
101152025-02-28T12:11:34.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 295 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 2c5e7e0f-563a-426b-b3cf-c3909e7d22ed response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
101162025-02-28T12:11:39.252ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::d]:54093
101172025-02-28T12:11:39.252ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = 104695a9-13a8-4f2e-b801-0c33e34fe6cd roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
101182025-02-28T12:11:39.253ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = 104695a9-13a8-4f2e-b801-0c33e34fe6cd resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
101192025-02-28T12:11:39.256ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3288 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = 104695a9-13a8-4f2e-b801-0c33e34fe6cd response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
101202025-02-28T12:11:39.256ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = 6a81ad1f-3fbf-4df3-8890-1e01e18846dc roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
101212025-02-28T12:11:39.256ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = 6a81ad1f-3fbf-4df3-8890-1e01e18846dc resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
101222025-02-28T12:11:39.258ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2687 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = 6a81ad1f-3fbf-4df3-8890-1e01e18846dc response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
101232025-02-28T12:11:40.452ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 26 reason = Timeout
101242025-02-28T12:11:40.452ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
101252025-02-28T12:11:40.452ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 26 reason = Timeout
101262025-02-28T12:11:40.452ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
101272025-02-28T12:11:40.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
101282025-02-28T12:11:40.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
101292025-02-28T12:11:40.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 76 reason = Timeout
101302025-02-28T12:11:40.455ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 26 reason = Timeout
101312025-02-28T12:11:40.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
101322025-02-28T12:11:40.455ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1535, tv_nsec: 93046189 } })
101332025-02-28T12:11:40.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
101342025-02-28T12:11:40.455ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
101352025-02-28T12:11:40.456ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1535, tv_nsec: 94023262 } })
101362025-02-28T12:11:40.456ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
101372025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101382025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 26 reason = Timeout
101392025-02-28T12:11:40.458ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
101402025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 26 reason = Timeout
101412025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
101422025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
101432025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 26 reason = Timeout
101442025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 26 reason = Timeout
101452025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
101462025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 28 reason = Timeout
101472025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
101482025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 76 reason = Timeout
101492025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
101502025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
101512025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 26 reason = Timeout
101522025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
101532025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 26 reason = Timeout
101542025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
101552025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 26 reason = Timeout
101562025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
101572025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
101582025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 26 reason = Timeout
101592025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
101602025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 26 reason = Timeout
101612025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
101622025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 26 reason = Timeout
101632025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
101642025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 26 reason = Timeout
101652025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
101662025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 26 reason = Timeout
101672025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
101682025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 26 reason = Timeout
101692025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
101702025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
101712025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
101722025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
101732025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
101742025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
101752025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
101762025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
101772025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
101782025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
101792025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
101802025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
101812025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
101822025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
101832025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
101842025-02-28T12:11:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
101852025-02-28T12:11:40.459ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.798772ms iteration = 26 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
101862025-02-28T12:11:40.460ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101872025-02-28T12:11:40.460ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101882025-02-28T12:11:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
101892025-02-28T12:11:40.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
101902025-02-28T12:11:40.461ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101912025-02-28T12:11:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
101922025-02-28T12:11:40.462ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
101932025-02-28T12:11:40.462ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
101942025-02-28T12:11:40.462ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101952025-02-28T12:11:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
101962025-02-28T12:11:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
101972025-02-28T12:11:40.463ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
101982025-02-28T12:11:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
101992025-02-28T12:11:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
102002025-02-28T12:11:40.464ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
102012025-02-28T12:11:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
102022025-02-28T12:11:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
102032025-02-28T12:11:40.464ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
102042025-02-28T12:11:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
102052025-02-28T12:11:40.465ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 7.339547ms iteration = 26 status = {"errors":[],"finish_invoked_ok":[]}
102062025-02-28T12:11:40.466ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102072025-02-28T12:11:40.467ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102082025-02-28T12:11:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
102092025-02-28T12:11:40.468ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
102102025-02-28T12:11:40.468ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
102112025-02-28T12:11:40.468ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
102122025-02-28T12:11:40.468ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
102132025-02-28T12:11:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
102142025-02-28T12:11:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
102152025-02-28T12:11:40.469ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
102162025-02-28T12:11:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
102172025-02-28T12:11:40.469ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
102182025-02-28T12:11:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
102192025-02-28T12:11:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
102202025-02-28T12:11:40.469ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
102212025-02-28T12:11:40.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
102222025-02-28T12:11:40.472ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
102232025-02-28T12:11:40.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
102242025-02-28T12:11:40.472ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
102252025-02-28T12:11:40.472ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1535, tv_nsec: 110103300 } })
102262025-02-28T12:11:40.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
102272025-02-28T12:11:40.472ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
102282025-02-28T12:11:40.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
102292025-02-28T12:11:40.473ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
102302025-02-28T12:11:40.473ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
102312025-02-28T12:11:40.473ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
102322025-02-28T12:11:40.473ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
102332025-02-28T12:11:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
102342025-02-28T12:11:40.473ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102352025-02-28T12:11:40.476ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
102362025-02-28T12:11:40.476ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
102372025-02-28T12:11:40.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
102382025-02-28T12:11:40.505ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
102392025-02-28T12:11:40.505ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
102402025-02-28T12:11:40.508ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
102412025-02-28T12:11:40.508ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 53.343408ms iteration = 76 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
102422025-02-28T12:11:40.511ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
102432025-02-28T12:11:40.512ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
102442025-02-28T12:11:40.513ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 54.824823ms iteration = 26 status = {"errors":[],"garbage_collect_requested":[]}
102452025-02-28T12:11:40.514ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 55.931251ms iteration = 26 status = {"errors":[],"requests_created_ok":[]}
102462025-02-28T12:11:40.514ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
102472025-02-28T12:11:40.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
102482025-02-28T12:11:40.516ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102492025-02-28T12:11:40.517ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
102502025-02-28T12:11:40.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
102512025-02-28T12:11:40.517ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102522025-02-28T12:11:40.517ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
102532025-02-28T12:11:40.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
102542025-02-28T12:11:40.518ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
102552025-02-28T12:11:40.518ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
102562025-02-28T12:11:40.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
102572025-02-28T12:11:40.520ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
102582025-02-28T12:11:40.520ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
102592025-02-28T12:11:40.520ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
102602025-02-28T12:11:40.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
102612025-02-28T12:11:40.520ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
102622025-02-28T12:11:40.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
102632025-02-28T12:11:40.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
102642025-02-28T12:11:40.521ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 62.948849ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
102652025-02-28T12:11:40.521ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 63.658067ms iteration = 26 status = 0
102662025-02-28T12:11:40.522ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 64.693342ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
102672025-02-28T12:11:40.523ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
102682025-02-28T12:11:40.523ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 65.639635ms iteration = 26 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
102692025-02-28T12:11:40.527ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102702025-02-28T12:11:40.528ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
102712025-02-28T12:11:40.528ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
102722025-02-28T12:11:40.528ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102732025-02-28T12:11:40.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
102742025-02-28T12:11:40.529ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
102752025-02-28T12:11:40.529ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
102762025-02-28T12:11:40.529ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 71.053466ms iteration = 26 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
102772025-02-28T12:11:40.530ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
102782025-02-28T12:11:40.530ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
102792025-02-28T12:11:40.530ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
102802025-02-28T12:11:40.539ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 81.137579ms iteration = 76 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
102812025-02-28T12:11:40.542ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
102822025-02-28T12:11:40.542ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
102832025-02-28T12:11:40.543ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec25c0d2-507b-46c9-acb1-2155093a15d1", "content-length": "26", "date": "Fri, 28 Feb 2025 12:11:40 GMT"} })
102842025-02-28T12:11:40.543ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
102852025-02-28T12:11:40.545ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
102862025-02-28T12:11:40.549ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
102872025-02-28T12:11:40.549ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
102882025-02-28T12:11:40.549ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f268f41-0263-4447-a574-b446618a15f3", "content-length": "26", "date": "Fri, 28 Feb 2025 12:11:40 GMT"} })
102892025-02-28T12:11:40.549ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
102902025-02-28T12:11:40.550ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 91.882578ms iteration = 26 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
102912025-02-28T12:11:40.551ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
102922025-02-28T12:11:40.559ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 101.359857ms iteration = 28 status = {"physical_disks_added":0}
102932025-02-28T12:11:40.561ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
102942025-02-28T12:11:40.561ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 102.860013ms iteration = 26 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
102952025-02-28T12:11:40.583ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
102962025-02-28T12:11:40.583ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
102972025-02-28T12:11:40.583ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9f996df-f853-47d7-b974-34977de98feb", "content-length": "26", "date": "Fri, 28 Feb 2025 12:11:40 GMT"} })
102982025-02-28T12:11:40.583ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
102992025-02-28T12:11:40.584ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
103002025-02-28T12:11:40.597ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
103012025-02-28T12:11:40.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
103022025-02-28T12:11:40.694ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 242.163376ms iteration = 26 status = {}
103032025-02-28T12:11:40.729ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
103042025-02-28T12:11:40.729ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
103052025-02-28T12:11:40.729ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "f47693bd-cb6c-436b-b26b-af7e18b491dc", "content-length": "372", "date": "Fri, 28 Feb 2025 12:11:40 GMT"} })
103062025-02-28T12:11:40.729ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
103072025-02-28T12:11:40.730ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
103082025-02-28T12:11:40.731ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103092025-02-28T12:11:40.734ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
103102025-02-28T12:11:40.734ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
103112025-02-28T12:11:40.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
103122025-02-28T12:11:40.735ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103132025-02-28T12:11:40.738ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
103142025-02-28T12:11:40.738ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
103152025-02-28T12:11:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
103162025-02-28T12:11:40.751ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
103172025-02-28T12:11:40.752ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "56a44eac-518a-4f7a-bed8-ae626eb7aca2", "content-length": "2", "date": "Fri, 28 Feb 2025 12:11:40 GMT"} })
103182025-02-28T12:11:40.754ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
103192025-02-28T12:11:40.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
103202025-02-28T12:11:40.756ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 304.411578ms iteration = 26 status = null
103212025-02-28T12:11:40.799ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
103222025-02-28T12:11:40.799ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
103232025-02-28T12:11:40.799ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "2782459d-3115-4447-a91c-121219e887e7", "date": "Fri, 28 Feb 2025 12:11:40 GMT"} })
103242025-02-28T12:11:40.799ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
103252025-02-28T12:11:40.800ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
103262025-02-28T12:11:40.802ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
103272025-02-28T12:11:40.802ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
103282025-02-28T12:11:40.806ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
103292025-02-28T12:11:40.809ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
103302025-02-28T12:11:40.813ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
103312025-02-28T12:11:40.813ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
103322025-02-28T12:11:40.814ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "d697789f-4c63-450d-8cfc-2eadeab76267", "date": "Fri, 28 Feb 2025 12:11:40 GMT"} })
103332025-02-28T12:11:40.815ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103342025-02-28T12:11:40.815ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103352025-02-28T12:11:40.816ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 358.194707ms iteration = 26 status = {}
103362025-02-28T12:11:40.818ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
103372025-02-28T12:11:40.818ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103382025-02-28T12:11:40.818ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103392025-02-28T12:11:40.828ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
103402025-02-28T12:11:40.829ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce12d7e1-1225-458b-bc59-5903f02f4f63", "content-length": "134", "date": "Fri, 28 Feb 2025 12:11:40 GMT"} })
103412025-02-28T12:11:40.829ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
103422025-02-28T12:11:40.829ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
103432025-02-28T12:11:40.829ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
103442025-02-28T12:11:40.829ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
103452025-02-28T12:11:40.829ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
103462025-02-28T12:11:40.829ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
103472025-02-28T12:11:40.829ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
103482025-02-28T12:11:40.829ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
103492025-02-28T12:11:40.829ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
103502025-02-28T12:11:40.829ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
103512025-02-28T12:11:40.829ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
103522025-02-28T12:11:40.829ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103532025-02-28T12:11:40.830ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103542025-02-28T12:11:40.832ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
103552025-02-28T12:11:40.832ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cc96b55-1590-4068-aa4b-9a9f537c8ef9", "content-length": "2", "date": "Fri, 28 Feb 2025 12:11:40 GMT"} })
103562025-02-28T12:11:40.832ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
103572025-02-28T12:11:40.832ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f2d0f9a-d7d8-4688-9c8d-5af5b856493a", "content-length": "51", "date": "Fri, 28 Feb 2025 12:11:40 GMT"} })
103582025-02-28T12:11:40.832ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
103592025-02-28T12:11:40.832ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
103602025-02-28T12:11:40.862ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
103612025-02-28T12:11:40.945ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "c3a618c9-c045-46c0-95d5-c2b72efadd88", "date": "Fri, 28 Feb 2025 12:11:40 GMT"} })
103622025-02-28T12:11:40.945ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
103632025-02-28T12:11:40.945ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
103642025-02-28T12:11:40.946ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b84f721-c63c-4c84-a139-2c7624849339", "content-length": "682", "date": "Fri, 28 Feb 2025 12:11:40 GMT"} })
103652025-02-28T12:11:40.946ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103662025-02-28T12:11:40.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103672025-02-28T12:11:40.951ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "28e1898c-6d06-44ae-9efb-99cd2f60fa55", "content-length": "691", "date": "Fri, 28 Feb 2025 12:11:40 GMT"} })
103682025-02-28T12:11:40.951ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
103692025-02-28T12:11:40.951ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
103702025-02-28T12:11:40.951ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
103712025-02-28T12:11:40.952ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103722025-02-28T12:11:40.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103732025-02-28T12:11:40.953ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 495.81908ms iteration = 26 status = {}
103742025-02-28T12:11:40.956ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103752025-02-28T12:11:40.957ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103762025-02-28T12:11:40.959ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103772025-02-28T12:11:40.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103782025-02-28T12:11:40.967ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 509.290745ms iteration = 26 status = {}
103792025-02-28T12:11:44.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 253 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 9cb7f4ac-247e-4260-abba-4672033a033e response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
103802025-02-28T12:11:50.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 77 reason = Timeout
103812025-02-28T12:11:50.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
103822025-02-28T12:11:50.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
103832025-02-28T12:11:50.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 77 reason = Timeout
103842025-02-28T12:11:50.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
103852025-02-28T12:11:50.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
103862025-02-28T12:11:50.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103872025-02-28T12:11:50.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
103882025-02-28T12:11:50.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
103892025-02-28T12:11:50.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
103902025-02-28T12:11:50.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
103912025-02-28T12:11:50.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
103922025-02-28T12:11:50.485ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
103932025-02-28T12:11:50.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
103942025-02-28T12:11:50.497ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103952025-02-28T12:11:50.499ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
103962025-02-28T12:11:50.499ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
103972025-02-28T12:11:50.500ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
103982025-02-28T12:11:50.512ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
103992025-02-28T12:11:50.512ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 57.039727ms iteration = 77 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
104002025-02-28T12:11:50.518ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 61.036674ms iteration = 77 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
104012025-02-28T12:11:54.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 277 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 1264b215-8c31-4e24-9c60-07ed85987857 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
104022025-02-28T12:12:00.456ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 78 reason = Timeout
104032025-02-28T12:12:00.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104042025-02-28T12:12:00.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104052025-02-28T12:12:00.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 78 reason = Timeout
104062025-02-28T12:12:00.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
104072025-02-28T12:12:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
104082025-02-28T12:12:00.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104092025-02-28T12:12:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
104102025-02-28T12:12:00.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104112025-02-28T12:12:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104122025-02-28T12:12:00.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104132025-02-28T12:12:00.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104142025-02-28T12:12:00.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104152025-02-28T12:12:00.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
104162025-02-28T12:12:00.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104172025-02-28T12:12:00.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104182025-02-28T12:12:00.476ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
104192025-02-28T12:12:00.476ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.041134ms iteration = 78 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
104202025-02-28T12:12:00.491ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
104212025-02-28T12:12:00.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
104222025-02-28T12:12:00.493ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 36.011969ms iteration = 78 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
104232025-02-28T12:12:04.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 258 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 1e5a7eef-1dca-4f52-b14d-634e14d9b8a4 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
104242025-02-28T12:12:09.253ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = a0838f90-7d75-4b27-aa47-341851e1394b roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
104252025-02-28T12:12:09.253ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = a0838f90-7d75-4b27-aa47-341851e1394b resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
104262025-02-28T12:12:09.255ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2440 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = a0838f90-7d75-4b27-aa47-341851e1394b response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
104272025-02-28T12:12:09.255ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = e2b38a02-ad66-4cfd-b6ae-6cb4a9c929b4 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
104282025-02-28T12:12:09.256ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = e2b38a02-ad66-4cfd-b6ae-6cb4a9c929b4 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
104292025-02-28T12:12:09.258ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2790 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = e2b38a02-ad66-4cfd-b6ae-6cb4a9c929b4 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
104302025-02-28T12:12:10.452ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 14 reason = Timeout
104312025-02-28T12:12:10.453ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-28 12:02:10.452957047 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
104322025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 14 reason = Timeout
104332025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
104342025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
104352025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
104362025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 27 reason = Timeout
104372025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 14 reason = Timeout
104382025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
104392025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 27 reason = Timeout
104402025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
104412025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
104422025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 16 reason = Timeout
104432025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 16 reason = Timeout
104442025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 14 reason = Timeout
104452025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
104462025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
104472025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 14 reason = Timeout
104482025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 14 reason = Timeout
104492025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
104502025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
104512025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
104522025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
104532025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
104542025-02-28T12:12:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
104552025-02-28T12:12:10.453ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5cefabf1-6831-46ff-98eb-1a3b5abad8b9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1565, tv_nsec: 91286056 } })
104562025-02-28T12:12:10.453ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5ebff2d9-48b6-4fe6-b852-684bb14b4955.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bddfb0f9-1d7b-45e4-b3a5-35aa6792956b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1565, tv_nsec: 91464030 } })
104572025-02-28T12:12:10.454ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
104582025-02-28T12:12:10.454ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 999.564\u{b5}s iteration = 14 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
104592025-02-28T12:12:10.454ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
104602025-02-28T12:12:10.454ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.2664ms iteration = 14 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
104612025-02-28T12:12:10.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 79 reason = Timeout
104622025-02-28T12:12:10.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 15 reason = Timeout
104632025-02-28T12:12:10.455ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
104642025-02-28T12:12:10.455ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 27 reason = Timeout
104652025-02-28T12:12:10.455ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 9.97\u{b5}s iteration = 15 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
104662025-02-28T12:12:10.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104672025-02-28T12:12:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104682025-02-28T12:12:10.455ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1565, tv_nsec: 93223503 } })
104692025-02-28T12:12:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
104702025-02-28T12:12:10.455ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
104712025-02-28T12:12:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
104722025-02-28T12:12:10.456ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104732025-02-28T12:12:10.456ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
104742025-02-28T12:12:10.456ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1565, tv_nsec: 94228767 } })
104752025-02-28T12:12:10.456ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
104762025-02-28T12:12:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
104772025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 14 reason = Timeout
104782025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 27 reason = Timeout
104792025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 27 reason = Timeout
104802025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 27 reason = Timeout
104812025-02-28T12:12:10.457ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
104822025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 27 reason = Timeout
104832025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
104842025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
104852025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
104862025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
104872025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 27 reason = Timeout
104882025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
104892025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 79 reason = Timeout
104902025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
104912025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
104922025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 14 reason = Timeout
104932025-02-28T12:12:10.457ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
104942025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
104952025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 29 reason = Timeout
104962025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 27 reason = Timeout
104972025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
104982025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
104992025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 27 reason = Timeout
105002025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
105012025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 27 reason = Timeout
105022025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
105032025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
105042025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
105052025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
105062025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
105072025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
105082025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
105092025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
105102025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
105112025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
105122025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
105132025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
105142025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
105152025-02-28T12:12:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
105162025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
105172025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
105182025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 27 reason = Timeout
105192025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
105202025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 27 reason = Timeout
105212025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
105222025-02-28T12:12:10.458ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-28 12:02:10.452957047 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
105232025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 14 reason = Timeout
105242025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.344418ms iteration = 14 status = {"expiration":"2025-02-28T12:02:10.452957047Z","pruned":[]}
105252025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
105262025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 14 reason = Timeout
105272025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
105282025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 27 reason = Timeout
105292025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
105302025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 27 reason = Timeout
105312025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 27 reason = Timeout
105322025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
105332025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
105342025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
105352025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
105362025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
105372025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
105382025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
105392025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
105402025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
105412025-02-28T12:12:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
105422025-02-28T12:12:10.460ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.679472ms iteration = 27 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
105432025-02-28T12:12:10.460ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105442025-02-28T12:12:10.460ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105452025-02-28T12:12:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
105462025-02-28T12:12:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
105472025-02-28T12:12:10.466ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105482025-02-28T12:12:10.467ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105492025-02-28T12:12:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
105502025-02-28T12:12:10.469ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
105512025-02-28T12:12:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
105522025-02-28T12:12:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
105532025-02-28T12:12:10.470ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
105542025-02-28T12:12:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
105552025-02-28T12:12:10.474ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105562025-02-28T12:12:10.474ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
105572025-02-28T12:12:10.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
105582025-02-28T12:12:10.477ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
105592025-02-28T12:12:10.477ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
105602025-02-28T12:12:10.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
105612025-02-28T12:12:10.479ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 21.407637ms iteration = 27 status = {"errors":[],"garbage_collect_requested":[]}
105622025-02-28T12:12:10.480ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 21.713064ms iteration = 14 status = {"errors":[],"found_port_ok":[]}
105632025-02-28T12:12:10.494ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105642025-02-28T12:12:10.499ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
105652025-02-28T12:12:10.499ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
105662025-02-28T12:12:10.499ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
105672025-02-28T12:12:10.513ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
105682025-02-28T12:12:10.513ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
105692025-02-28T12:12:10.513ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
105702025-02-28T12:12:10.513ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
105712025-02-28T12:12:10.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
105722025-02-28T12:12:10.514ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105732025-02-28T12:12:10.515ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
105742025-02-28T12:12:10.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
105752025-02-28T12:12:10.516ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
105762025-02-28T12:12:10.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
105772025-02-28T12:12:10.516ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
105782025-02-28T12:12:10.516ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
105792025-02-28T12:12:10.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
105802025-02-28T12:12:10.517ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105812025-02-28T12:12:10.517ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 59.305132ms iteration = 27 status = {"errors":[],"finish_invoked_ok":[]}
105822025-02-28T12:12:10.519ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
105832025-02-28T12:12:10.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
105842025-02-28T12:12:10.521ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
105852025-02-28T12:12:10.521ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
105862025-02-28T12:12:10.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
105872025-02-28T12:12:10.523ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
105882025-02-28T12:12:10.523ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
105892025-02-28T12:12:10.523ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
105902025-02-28T12:12:10.523ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
105912025-02-28T12:12:10.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
105922025-02-28T12:12:10.524ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105932025-02-28T12:12:10.525ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
105942025-02-28T12:12:10.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
105952025-02-28T12:12:10.525ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105962025-02-28T12:12:10.526ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
105972025-02-28T12:12:10.526ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
105982025-02-28T12:12:10.526ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
105992025-02-28T12:12:10.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
106002025-02-28T12:12:10.526ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106012025-02-28T12:12:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
106022025-02-28T12:12:10.528ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
106032025-02-28T12:12:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
106042025-02-28T12:12:10.528ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
106052025-02-28T12:12:10.528ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
106062025-02-28T12:12:10.529ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
106072025-02-28T12:12:10.529ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
106082025-02-28T12:12:10.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
106092025-02-28T12:12:10.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
106102025-02-28T12:12:10.530ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106112025-02-28T12:12:10.532ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
106122025-02-28T12:12:10.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
106132025-02-28T12:12:10.533ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
106142025-02-28T12:12:10.533ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
106152025-02-28T12:12:10.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
106162025-02-28T12:12:10.534ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
106172025-02-28T12:12:10.534ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
106182025-02-28T12:12:10.534ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 76.137228ms iteration = 14 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
106192025-02-28T12:12:10.535ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106202025-02-28T12:12:10.537ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
106212025-02-28T12:12:10.537ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
106222025-02-28T12:12:10.537ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
106232025-02-28T12:12:10.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
106242025-02-28T12:12:10.539ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
106252025-02-28T12:12:10.539ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
106262025-02-28T12:12:10.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
106272025-02-28T12:12:10.542ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
106282025-02-28T12:12:10.542ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 85.243598ms iteration = 27 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
106292025-02-28T12:12:10.543ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 84.776837ms iteration = 27 status = {"errors":[],"requests_created_ok":[]}
106302025-02-28T12:12:10.543ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1565, tv_nsec: 181296139 } })
106312025-02-28T12:12:10.546ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
106322025-02-28T12:12:10.546ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
106332025-02-28T12:12:10.546ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 89.47741ms iteration = 14 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
106342025-02-28T12:12:10.547ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
106352025-02-28T12:12:10.547ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106362025-02-28T12:12:10.547ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
106372025-02-28T12:12:10.547ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
106382025-02-28T12:12:10.547ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
106392025-02-28T12:12:10.548ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
106402025-02-28T12:12:10.551ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
106412025-02-28T12:12:10.551ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
106422025-02-28T12:12:10.551ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
106432025-02-28T12:12:10.553ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
106442025-02-28T12:12:10.553ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106452025-02-28T12:12:10.553ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 95.552607ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
106462025-02-28T12:12:10.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
106472025-02-28T12:12:10.555ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
106482025-02-28T12:12:10.555ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
106492025-02-28T12:12:10.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
106502025-02-28T12:12:10.563ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
106512025-02-28T12:12:10.563ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 107.726331ms iteration = 79 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
106522025-02-28T12:12:10.565ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106532025-02-28T12:12:10.565ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
106542025-02-28T12:12:10.565ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
106552025-02-28T12:12:10.568ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 111.388809ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
106562025-02-28T12:12:10.570ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
106572025-02-28T12:12:10.570ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
106582025-02-28T12:12:10.570ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
106592025-02-28T12:12:10.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
106602025-02-28T12:12:10.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
106612025-02-28T12:12:10.572ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
106622025-02-28T12:12:10.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
106632025-02-28T12:12:10.573ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
106642025-02-28T12:12:10.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
106652025-02-28T12:12:10.573ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
106662025-02-28T12:12:10.573ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 0 version = 2
106672025-02-28T12:12:10.573ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
106682025-02-28T12:12:10.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
106692025-02-28T12:12:10.574ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
106702025-02-28T12:12:10.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
106712025-02-28T12:12:10.575ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106722025-02-28T12:12:10.576ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106732025-02-28T12:12:10.577ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
106742025-02-28T12:12:10.577ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
106752025-02-28T12:12:10.578ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
106762025-02-28T12:12:10.578ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
106772025-02-28T12:12:10.578ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
106782025-02-28T12:12:10.578ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
106792025-02-28T12:12:10.578ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
106802025-02-28T12:12:10.579ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 122.622411ms iteration = 79 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
106812025-02-28T12:12:10.580ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106822025-02-28T12:12:10.580ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 123.357208ms iteration = 27 status = 0
106832025-02-28T12:12:10.580ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
106842025-02-28T12:12:10.580ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
106852025-02-28T12:12:10.581ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
106862025-02-28T12:12:10.583ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 1 version = 2
106872025-02-28T12:12:10.583ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
106882025-02-28T12:12:10.583ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-02-28 11:59:52.542241 UTC dns_group = external generation = 2
106892025-02-28T12:12:10.583ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 129.923946ms iteration = 14 status = {"generation":2}
106902025-02-28T12:12:10.583ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
106912025-02-28T12:12:10.583ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
106922025-02-28T12:12:10.584ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
106932025-02-28T12:12:10.584ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
106942025-02-28T12:12:10.584ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
106952025-02-28T12:12:10.585ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106962025-02-28T12:12:10.585ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
106972025-02-28T12:12:10.585ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
106982025-02-28T12:12:10.585ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 128.528063ms iteration = 27 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
106992025-02-28T12:12:10.587ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
107002025-02-28T12:12:10.587ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
107012025-02-28T12:12:10.587ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-02-28 11:59:51.915187 UTC dns_group = internal generation = 1
107022025-02-28T12:12:10.587ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 134.175419ms iteration = 14 status = {"generation":1}
107032025-02-28T12:12:10.587ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
107042025-02-28T12:12:10.587ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
107052025-02-28T12:12:10.587ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
107062025-02-28T12:12:10.590ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
107072025-02-28T12:12:10.590ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
107082025-02-28T12:12:10.590ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "307ebd4d-b435-454f-addc-cda9bf96ccc7", "content-length": "26", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} })
107092025-02-28T12:12:10.590ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
107102025-02-28T12:12:10.593ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
107112025-02-28T12:12:10.593ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
107122025-02-28T12:12:10.593ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
107132025-02-28T12:12:10.593ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
107142025-02-28T12:12:10.594ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8417810-7809-4249-a689-32c63d1e1896", "content-length": "26", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} })
107152025-02-28T12:12:10.594ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
107162025-02-28T12:12:10.596ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107172025-02-28T12:12:10.597ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
107182025-02-28T12:12:10.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
107192025-02-28T12:12:10.599ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
107202025-02-28T12:12:10.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
107212025-02-28T12:12:10.606ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 148.120436ms iteration = 27 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
107222025-02-28T12:12:10.628ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
107232025-02-28T12:12:10.656ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
107242025-02-28T12:12:10.657ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 199.807503ms iteration = 27 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
107252025-02-28T12:12:10.675ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-28T11:59:08.719399Z, time_modified: 2025-02-28T11:59:08.719399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }) }
107262025-02-28T12:12:10.675ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 222.712606ms iteration = 14 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]}},"default_endpoint":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
107272025-02-28T12:12:10.693ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
107282025-02-28T12:12:10.693ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 236.5676ms iteration = 14 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
107292025-02-28T12:12:10.701ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 244.791619ms iteration = 29 status = {"physical_disks_added":0}
107302025-02-28T12:12:10.737ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
107312025-02-28T12:12:10.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
107322025-02-28T12:12:10.747ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107332025-02-28T12:12:10.752ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
107342025-02-28T12:12:10.752ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
107352025-02-28T12:12:10.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
107362025-02-28T12:12:10.755ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107372025-02-28T12:12:10.758ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
107382025-02-28T12:12:10.758ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
107392025-02-28T12:12:10.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
107402025-02-28T12:12:10.813ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
107412025-02-28T12:12:10.814ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "b715d9f5-796d-4de7-a09f-fc9b8276b888", "content-length": "2", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} })
107422025-02-28T12:12:10.815ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
107432025-02-28T12:12:10.816ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
107442025-02-28T12:12:10.823ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 370.145335ms iteration = 27 status = null
107452025-02-28T12:12:10.857ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
107462025-02-28T12:12:10.858ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "1cd0a3d6-1205-475c-b113-5ae7512660c1", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
107472025-02-28T12:12:10.858ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
107482025-02-28T12:12:10.859ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "aeef0a1a-865d-4035-b1c3-eb25606e7541", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
107492025-02-28T12:12:10.860ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
107502025-02-28T12:12:10.861ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
107512025-02-28T12:12:10.861ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 408.655345ms iteration = 16 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
107522025-02-28T12:12:10.864ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
107532025-02-28T12:12:10.864ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
107542025-02-28T12:12:10.864ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "62d6bf3b-6617-4bf1-b3e1-fc9c3ccc8afe", "content-length": "26", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} })
107552025-02-28T12:12:10.864ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
107562025-02-28T12:12:10.865ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 412.856566ms iteration = 27 status = {}
107572025-02-28T12:12:10.867ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
107582025-02-28T12:12:10.904ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
107592025-02-28T12:12:10.904ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
107602025-02-28T12:12:10.904ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3317d8e5-ef7e-486f-8f13-7357f907eb41", "content-length": "372", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} })
107612025-02-28T12:12:10.904ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
107622025-02-28T12:12:10.905ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
107632025-02-28T12:12:10.930ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
107642025-02-28T12:12:10.931ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
107652025-02-28T12:12:10.931ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c7cd206-ba82-438a-b343-fc91e41e5df7", "content-length": "691", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} })
107662025-02-28T12:12:10.931ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
107672025-02-28T12:12:10.931ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
107682025-02-28T12:12:10.931ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
107692025-02-28T12:12:10.934ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 476.923773ms iteration = 27 status = {}
107702025-02-28T12:12:10.956ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
107712025-02-28T12:12:10.956ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
107722025-02-28T12:12:10.956ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "61c7efed-52df-447f-b6af-a92783a72abd", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} })
107732025-02-28T12:12:10.956ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
107742025-02-28T12:12:10.957ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
107752025-02-28T12:12:10.959ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
107762025-02-28T12:12:10.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
107772025-02-28T12:12:10.962ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
107782025-02-28T12:12:10.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
107792025-02-28T12:12:10.966ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
107802025-02-28T12:12:10.966ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
107812025-02-28T12:12:10.966ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "5c0205c4-d0cc-498b-8d17-89c6dfe2541a", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} })
107822025-02-28T12:12:10.968ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 511.771384ms iteration = 27 status = {}
107832025-02-28T12:12:11.012ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
107842025-02-28T12:12:11.012ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "395282c2-9501-40a6-bfa0-34962aeaa991", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
107852025-02-28T12:12:11.012ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
107862025-02-28T12:12:11.014ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "45229b8e-b846-4f17-ad1e-48dc768a1f69", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
107872025-02-28T12:12:11.014ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
107882025-02-28T12:12:11.015ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "1a64a7af-b391-4e46-be5f-0f48e4c72e81", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
107892025-02-28T12:12:11.015ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
107902025-02-28T12:12:11.017ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
107912025-02-28T12:12:11.017ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 564.529928ms iteration = 16 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
107922025-02-28T12:12:11.046ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
107932025-02-28T12:12:11.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
107942025-02-28T12:12:11.048ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
107952025-02-28T12:12:11.048ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
107962025-02-28T12:12:11.049ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
107972025-02-28T12:12:11.058ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
107982025-02-28T12:12:11.058ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "4132535e-7420-416a-88f5-0a934f6d9fea", "content-length": "134", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} })
107992025-02-28T12:12:11.058ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
108002025-02-28T12:12:11.058ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
108012025-02-28T12:12:11.058ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
108022025-02-28T12:12:11.058ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
108032025-02-28T12:12:11.058ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
108042025-02-28T12:12:11.058ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
108052025-02-28T12:12:11.058ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
108062025-02-28T12:12:11.058ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
108072025-02-28T12:12:11.059ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
108082025-02-28T12:12:11.059ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
108092025-02-28T12:12:11.059ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
108102025-02-28T12:12:11.059ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
108112025-02-28T12:12:11.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
108122025-02-28T12:12:11.061ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
108132025-02-28T12:12:11.061ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "5594b518-5b40-429c-b9fa-e45aa0ef2480", "content-length": "2", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} })
108142025-02-28T12:12:11.061ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
108152025-02-28T12:12:11.061ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "309c0f18-6665-44bb-a9f5-ab5e10688cf1", "content-length": "51", "date": "Fri, 28 Feb 2025 12:12:10 GMT"} })
108162025-02-28T12:12:11.061ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
108172025-02-28T12:12:11.061ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
108182025-02-28T12:12:11.217ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "357fd8f0-4bbe-4e8a-9da8-f481c373f98d", "date": "Fri, 28 Feb 2025 12:12:11 GMT"} })
108192025-02-28T12:12:11.217ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
108202025-02-28T12:12:11.217ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
108212025-02-28T12:12:11.217ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4554a8b-6df0-4c3b-a80f-67ff676a772f", "content-length": "682", "date": "Fri, 28 Feb 2025 12:12:11 GMT"} })
108222025-02-28T12:12:11.217ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
108232025-02-28T12:12:11.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
108242025-02-28T12:12:11.224ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
108252025-02-28T12:12:11.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
108262025-02-28T12:12:11.227ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
108272025-02-28T12:12:11.228ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
108282025-02-28T12:12:11.229ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
108292025-02-28T12:12:11.230ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
108302025-02-28T12:12:11.237ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 780.769888ms iteration = 27 status = {}
108312025-02-28T12:12:14.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 301 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 4437abf8-61b9-4a68-b4ff-3fd896d6bf4e response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
108322025-02-28T12:12:20.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 80 reason = Timeout
108332025-02-28T12:12:20.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
108342025-02-28T12:12:20.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
108352025-02-28T12:12:20.458ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 80 reason = Timeout
108362025-02-28T12:12:20.458ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
108372025-02-28T12:12:20.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
108382025-02-28T12:12:20.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108392025-02-28T12:12:20.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
108402025-02-28T12:12:20.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
108412025-02-28T12:12:20.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
108422025-02-28T12:12:20.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
108432025-02-28T12:12:20.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
108442025-02-28T12:12:20.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108452025-02-28T12:12:20.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
108462025-02-28T12:12:20.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
108472025-02-28T12:12:20.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
108482025-02-28T12:12:20.482ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
108492025-02-28T12:12:20.482ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 27.061793ms iteration = 80 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
108502025-02-28T12:12:20.490ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
108512025-02-28T12:12:20.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
108522025-02-28T12:12:20.492ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.667347ms iteration = 80 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
108532025-02-28T12:12:24.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 420 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = e36dc48d-a7ff-427f-9cef-218fe4598e87 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
108542025-02-28T12:12:30.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 81 reason = Timeout
108552025-02-28T12:12:30.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
108562025-02-28T12:12:30.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
108572025-02-28T12:12:30.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 81 reason = Timeout
108582025-02-28T12:12:30.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
108592025-02-28T12:12:30.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
108602025-02-28T12:12:30.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108612025-02-28T12:12:30.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
108622025-02-28T12:12:30.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
108632025-02-28T12:12:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
108642025-02-28T12:12:30.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
108652025-02-28T12:12:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
108662025-02-28T12:12:30.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
108672025-02-28T12:12:30.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
108682025-02-28T12:12:30.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108692025-02-28T12:12:30.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
108702025-02-28T12:12:30.468ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
108712025-02-28T12:12:30.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
108722025-02-28T12:12:30.481ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
108732025-02-28T12:12:30.481ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.604362ms iteration = 81 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
108742025-02-28T12:12:30.497ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 40.17419ms iteration = 81 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
108752025-02-28T12:12:34.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 294 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 83c444a5-8565-41d7-8e97-bd2d9adc70b2 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
108762025-02-28T12:12:39.252ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = 34e03dd6-d39f-4aca-83bf-07f652639ada roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
108772025-02-28T12:12:39.253ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = 34e03dd6-d39f-4aca-83bf-07f652639ada resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
108782025-02-28T12:12:39.256ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3362 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = 34e03dd6-d39f-4aca-83bf-07f652639ada response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
108792025-02-28T12:12:39.256ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = f84821df-c9fb-44e1-b712-96ca3da44c21 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
108802025-02-28T12:12:39.256ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = f84821df-c9fb-44e1-b712-96ca3da44c21 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
108812025-02-28T12:12:39.259ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2604 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54093 req_id = f84821df-c9fb-44e1-b712-96ca3da44c21 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
108822025-02-28T12:12:40.452ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 28 reason = Timeout
108832025-02-28T12:12:40.452ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 28 reason = Timeout
108842025-02-28T12:12:40.452ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
108852025-02-28T12:12:40.452ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
108862025-02-28T12:12:40.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
108872025-02-28T12:12:40.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
108882025-02-28T12:12:40.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 82 reason = Timeout
108892025-02-28T12:12:40.455ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 28 reason = Timeout
108902025-02-28T12:12:40.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
108912025-02-28T12:12:40.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
108922025-02-28T12:12:40.455ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1595, tv_nsec: 93240503 } })
108932025-02-28T12:12:40.455ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1595, tv_nsec: 93412167 } })
108942025-02-28T12:12:40.455ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
108952025-02-28T12:12:40.455ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
108962025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 82 reason = Timeout
108972025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 28 reason = Timeout
108982025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
108992025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
109002025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 28 reason = Timeout
109012025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
109022025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
109032025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 30 reason = Timeout
109042025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
109052025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 28 reason = Timeout
109062025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 28 reason = Timeout
109072025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
109082025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 28 reason = Timeout
109092025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109102025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
109112025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 28 reason = Timeout
109122025-02-28T12:12:40.457ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
109132025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109142025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 28 reason = Timeout
109152025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
109162025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109172025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 28 reason = Timeout
109182025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
109192025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
109202025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
109212025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
109222025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
109232025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109242025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
109252025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
109262025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
109272025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109282025-02-28T12:12:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
109292025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 28 reason = Timeout
109302025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 28 reason = Timeout
109312025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
109322025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
109332025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.173177ms iteration = 28 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
109342025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 28 reason = Timeout
109352025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
109362025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 28 reason = Timeout
109372025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
109382025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 28 reason = Timeout
109392025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
109402025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
109412025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
109422025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
109432025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
109442025-02-28T12:12:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
109452025-02-28T12:12:40.459ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109462025-02-28T12:12:40.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
109472025-02-28T12:12:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
109482025-02-28T12:12:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
109492025-02-28T12:12:40.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
109502025-02-28T12:12:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
109512025-02-28T12:12:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
109522025-02-28T12:12:40.461ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
109532025-02-28T12:12:40.461ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109542025-02-28T12:12:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
109552025-02-28T12:12:40.462ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109562025-02-28T12:12:40.463ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109572025-02-28T12:12:40.463ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
109582025-02-28T12:12:40.464ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109592025-02-28T12:12:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109602025-02-28T12:12:40.464ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109612025-02-28T12:12:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
109622025-02-28T12:12:40.464ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
109632025-02-28T12:12:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
109642025-02-28T12:12:40.465ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109652025-02-28T12:12:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
109662025-02-28T12:12:40.465ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
109672025-02-28T12:12:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
109682025-02-28T12:12:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
109692025-02-28T12:12:40.467ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109702025-02-28T12:12:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
109712025-02-28T12:12:40.467ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
109722025-02-28T12:12:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109732025-02-28T12:12:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
109742025-02-28T12:12:40.469ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 11.278742ms iteration = 28 status = {"errors":[],"requests_created_ok":[]}
109752025-02-28T12:12:40.470ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 13.081076ms iteration = 82 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
109762025-02-28T12:12:40.471ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
109772025-02-28T12:12:40.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
109782025-02-28T12:12:40.472ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109792025-02-28T12:12:40.472ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
109802025-02-28T12:12:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
109812025-02-28T12:12:40.473ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109822025-02-28T12:12:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109832025-02-28T12:12:40.474ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109842025-02-28T12:12:40.475ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109852025-02-28T12:12:40.475ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
109862025-02-28T12:12:40.475ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109872025-02-28T12:12:40.475ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1595, tv_nsec: 113199835 } })
109882025-02-28T12:12:40.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109892025-02-28T12:12:40.475ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
109902025-02-28T12:12:40.479ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
109912025-02-28T12:12:40.479ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
109922025-02-28T12:12:40.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
109932025-02-28T12:12:40.479ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
109942025-02-28T12:12:40.479ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
109952025-02-28T12:12:40.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
109962025-02-28T12:12:40.501ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 44.055753ms iteration = 30 status = {"physical_disks_added":0}
109972025-02-28T12:12:40.505ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
109982025-02-28T12:12:40.505ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 50.663083ms iteration = 82 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
109992025-02-28T12:12:40.510ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
110002025-02-28T12:12:40.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
110012025-02-28T12:12:40.512ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110022025-02-28T12:12:40.512ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110032025-02-28T12:12:40.513ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
110042025-02-28T12:12:40.513ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
110052025-02-28T12:12:40.513ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
110062025-02-28T12:12:40.513ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
110072025-02-28T12:12:40.515ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 56.896244ms iteration = 28 status = {"errors":[],"finish_invoked_ok":[]}
110082025-02-28T12:12:40.517ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 59.078876ms iteration = 28 status = {"errors":[],"garbage_collect_requested":[]}
110092025-02-28T12:12:40.525ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
110102025-02-28T12:12:40.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
110112025-02-28T12:12:40.525ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
110122025-02-28T12:12:40.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
110132025-02-28T12:12:40.526ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
110142025-02-28T12:12:40.526ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 69.277952ms iteration = 28 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
110152025-02-28T12:12:40.526ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
110162025-02-28T12:12:40.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
110172025-02-28T12:12:40.561ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
110182025-02-28T12:12:40.561ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
110192025-02-28T12:12:40.562ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdf05a7f-8391-42a4-822f-ead69c47caf5", "content-length": "26", "date": "Fri, 28 Feb 2025 12:12:40 GMT"} })
110202025-02-28T12:12:40.562ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
110212025-02-28T12:12:40.564ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
110222025-02-28T12:12:40.566ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
110232025-02-28T12:12:40.566ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
110242025-02-28T12:12:40.566ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa619976-d895-4c17-9315-83cbd4467bab", "content-length": "26", "date": "Fri, 28 Feb 2025 12:12:40 GMT"} })
110252025-02-28T12:12:40.566ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
110262025-02-28T12:12:40.568ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
110272025-02-28T12:12:40.584ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
110282025-02-28T12:12:40.584ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
110292025-02-28T12:12:40.596ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
110302025-02-28T12:12:40.596ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
110312025-02-28T12:12:40.597ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ff129f4-13d9-4010-9711-8218dec63eaa", "content-length": "372", "date": "Fri, 28 Feb 2025 12:12:40 GMT"} })
110322025-02-28T12:12:40.597ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
110332025-02-28T12:12:40.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
110342025-02-28T12:12:40.599ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 142.395057ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
110352025-02-28T12:12:40.601ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110362025-02-28T12:12:40.601ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
110372025-02-28T12:12:40.601ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
110382025-02-28T12:12:40.601ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
110392025-02-28T12:12:40.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
110402025-02-28T12:12:40.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110412025-02-28T12:12:40.609ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
110422025-02-28T12:12:40.609ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
110432025-02-28T12:12:40.609ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "617d54b4-78bc-4aa7-b541-02f1c4ea9521", "content-length": "26", "date": "Fri, 28 Feb 2025 12:12:40 GMT"} })
110442025-02-28T12:12:40.609ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
110452025-02-28T12:12:40.611ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
110462025-02-28T12:12:40.640ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
110472025-02-28T12:12:40.640ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
110482025-02-28T12:12:40.640ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "b8f9b76e-fb59-451d-a260-1f5f69b9689b", "date": "Fri, 28 Feb 2025 12:12:40 GMT"} })
110492025-02-28T12:12:40.640ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
110502025-02-28T12:12:40.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
110512025-02-28T12:12:40.644ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
110522025-02-28T12:12:40.644ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
110532025-02-28T12:12:40.648ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
110542025-02-28T12:12:40.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
110552025-02-28T12:12:40.654ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
110562025-02-28T12:12:40.655ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
110572025-02-28T12:12:40.656ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "652d8bfd-7364-43bb-988f-e86e6045a7ab", "date": "Fri, 28 Feb 2025 12:12:40 GMT"} })
110582025-02-28T12:12:40.657ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 200.844268ms iteration = 28 status = {}
110592025-02-28T12:12:40.658ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110602025-02-28T12:12:40.661ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
110612025-02-28T12:12:40.661ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110622025-02-28T12:12:40.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110632025-02-28T12:12:40.663ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110642025-02-28T12:12:40.663ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 206.222287ms iteration = 28 status = 0
110652025-02-28T12:12:40.663ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
110662025-02-28T12:12:40.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
110672025-02-28T12:12:40.665ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
110682025-02-28T12:12:40.665ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "f51f4d78-15b1-4913-80e6-3b5691d1f629", "content-length": "2", "date": "Fri, 28 Feb 2025 12:12:39 GMT"} })
110692025-02-28T12:12:40.665ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
110702025-02-28T12:12:40.666ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
110712025-02-28T12:12:40.666ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
110722025-02-28T12:12:40.667ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
110732025-02-28T12:12:40.667ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 210.294106ms iteration = 28 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
110742025-02-28T12:12:40.667ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
110752025-02-28T12:12:40.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
110762025-02-28T12:12:40.669ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 211.374442ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
110772025-02-28T12:12:40.669ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 217.092421ms iteration = 28 status = null
110782025-02-28T12:12:40.670ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
110792025-02-28T12:12:40.670ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
110802025-02-28T12:12:40.670ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 213.605546ms iteration = 28 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
110812025-02-28T12:12:40.677ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 225.443871ms iteration = 28 status = {}
110822025-02-28T12:12:40.696ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 237.967374ms iteration = 28 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
110832025-02-28T12:12:40.736ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
110842025-02-28T12:12:40.736ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
110852025-02-28T12:12:40.738ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110862025-02-28T12:12:40.741ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
110872025-02-28T12:12:40.741ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
110882025-02-28T12:12:40.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
110892025-02-28T12:12:40.743ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110902025-02-28T12:12:40.745ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
110912025-02-28T12:12:40.745ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
110922025-02-28T12:12:40.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
110932025-02-28T12:12:40.769ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
110942025-02-28T12:12:40.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
110952025-02-28T12:12:40.772ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
110962025-02-28T12:12:40.772ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
110972025-02-28T12:12:40.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
110982025-02-28T12:12:40.782ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
110992025-02-28T12:12:40.782ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb9a7c87-669e-4943-be88-509ec05011f3", "content-length": "134", "date": "Fri, 28 Feb 2025 12:12:39 GMT"} })
111002025-02-28T12:12:40.782ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
111012025-02-28T12:12:40.782ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
111022025-02-28T12:12:40.783ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
111032025-02-28T12:12:40.783ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
111042025-02-28T12:12:40.783ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
111052025-02-28T12:12:40.783ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
111062025-02-28T12:12:40.783ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
111072025-02-28T12:12:40.783ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
111082025-02-28T12:12:40.783ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
111092025-02-28T12:12:40.783ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
111102025-02-28T12:12:40.783ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
111112025-02-28T12:12:40.783ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
111122025-02-28T12:12:40.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
111132025-02-28T12:12:40.787ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
111142025-02-28T12:12:40.787ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef62e9a6-6e17-4f49-9d4d-7f0ea879aaf8", "content-length": "2", "date": "Fri, 28 Feb 2025 12:12:39 GMT"} })
111152025-02-28T12:12:40.787ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
111162025-02-28T12:12:40.787ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cd91a10-5e9f-4ca2-996b-298b60c33a0e", "content-length": "51", "date": "Fri, 28 Feb 2025 12:12:39 GMT"} })
111172025-02-28T12:12:40.787ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
111182025-02-28T12:12:40.787ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
111192025-02-28T12:12:40.798ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
111202025-02-28T12:12:40.899ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "fb35a024-95a9-4421-9181-a8278425f1bd", "date": "Fri, 28 Feb 2025 12:12:40 GMT"} })
111212025-02-28T12:12:40.899ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
111222025-02-28T12:12:40.899ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
111232025-02-28T12:12:40.900ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d06abf4-b3cf-4eba-920f-ae1eefaa77d8", "content-length": "682", "date": "Fri, 28 Feb 2025 12:12:40 GMT"} })
111242025-02-28T12:12:40.900ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
111252025-02-28T12:12:40.900ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
111262025-02-28T12:12:40.900ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfba701a-e337-40d7-a915-106d73dff485", "content-length": "691", "date": "Fri, 28 Feb 2025 12:12:40 GMT"} })
111272025-02-28T12:12:40.900ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
111282025-02-28T12:12:40.900ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
111292025-02-28T12:12:40.900ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
111302025-02-28T12:12:40.902ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 445.513115ms iteration = 28 status = {}
111312025-02-28T12:12:40.905ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
111322025-02-28T12:12:40.905ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
111332025-02-28T12:12:40.910ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
111342025-02-28T12:12:40.911ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
111352025-02-28T12:12:40.914ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
111362025-02-28T12:12:40.914ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
111372025-02-28T12:12:40.921ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 464.556485ms iteration = 28 status = {}
111382025-02-28T12:12:41.429ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
111392025-02-28T12:12:41.429ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::a]:12221
111402025-02-28T12:12:41.459ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
111412025-02-28T12:12:41.459ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::a]:64017
111422025-02-28T12:12:41.459ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:64017 req_id = 1f8638aa-25b0-483d-9440-5c0ad4123066 roles = RoleSet { roles: {} } uri = /metrics/producers
111432025-02-28T12:12:41.459ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:64017 req_id = 1f8638aa-25b0-483d-9440-5c0ad4123066 resource = Database result = Ok(()) uri = /metrics/producers
111442025-02-28T12:12:41.500ZINFOnexus: assigned collector to new producer collector_id = a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79 file = nexus/src/app/oximeter.rs:76 producer_id = c1b34bc5-1c2f-4d3c-808c-b7a441aac295
111452025-02-28T12:12:41.500ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 40973 local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:64017 req_id = 1f8638aa-25b0-483d-9440-5c0ad4123066 response_code = 201 uri = /metrics/producers
111462025-02-28T12:12:41.500ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "1f8638aa-25b0-483d-9440-5c0ad4123066", "content-length": "41", "date": "Fri, 28 Feb 2025 12:12:41 GMT"} })
111472025-02-28T12:12:41.501ZDEBGnexus (producer-registration-task): registered with nexus successfully
111482025-02-28T12:12:41.501ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
111492025-02-28T12:12:44.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 390 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = d12cd020-5613-41e1-949c-78bd52553762 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
111502025-02-28T12:12:50.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 83 reason = Timeout
111512025-02-28T12:12:50.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
111522025-02-28T12:12:50.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
111532025-02-28T12:12:50.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 83 reason = Timeout
111542025-02-28T12:12:50.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
111552025-02-28T12:12:50.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
111562025-02-28T12:12:50.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111572025-02-28T12:12:50.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
111582025-02-28T12:12:50.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
111592025-02-28T12:12:50.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
111602025-02-28T12:12:50.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
111612025-02-28T12:12:50.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
111622025-02-28T12:12:50.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111632025-02-28T12:12:50.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
111642025-02-28T12:12:50.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
111652025-02-28T12:12:50.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
111662025-02-28T12:12:50.480ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
111672025-02-28T12:12:50.480ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.326237ms iteration = 83 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
111682025-02-28T12:12:50.485ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
111692025-02-28T12:12:50.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
111702025-02-28T12:12:50.516ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.455065ms iteration = 83 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
111712025-02-28T12:12:54.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 309 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 0c1ccbd7-0a1b-448b-be11-e2a26b22b5c4 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
111722025-02-28T12:13:00.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 84 reason = Timeout
111732025-02-28T12:13:00.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
111742025-02-28T12:13:00.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
111752025-02-28T12:13:00.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111762025-02-28T12:13:00.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 84 reason = Timeout
111772025-02-28T12:13:00.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
111782025-02-28T12:13:00.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
111792025-02-28T12:13:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
111802025-02-28T12:13:00.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
111812025-02-28T12:13:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
111822025-02-28T12:13:00.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
111832025-02-28T12:13:00.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
111842025-02-28T12:13:00.463ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111852025-02-28T12:13:00.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
111862025-02-28T12:13:00.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
111872025-02-28T12:13:00.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
111882025-02-28T12:13:00.479ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
111892025-02-28T12:13:00.479ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.943148ms iteration = 84 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
111902025-02-28T12:13:00.496ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
111912025-02-28T12:13:00.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
111922025-02-28T12:13:00.498ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 40.974109ms iteration = 84 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
111932025-02-28T12:13:04.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 301 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 2112d0f1-d5cc-41db-8ca7-fd7fcc8672ac response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
111942025-02-28T12:13:10.452ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 15 reason = Timeout
111952025-02-28T12:13:10.452ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 15 reason = Timeout
111962025-02-28T12:13:10.452ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-28 12:03:10.452823850 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
111972025-02-28T12:13:10.452ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
111982025-02-28T12:13:10.452ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
111992025-02-28T12:13:10.452ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
112002025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 29 reason = Timeout
112012025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
112022025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 17 reason = Timeout
112032025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 15 reason = Timeout
112042025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
112052025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
112062025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 29 reason = Timeout
112072025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
112082025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 17 reason = Timeout
112092025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 15 reason = Timeout
112102025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 15 reason = Timeout
112112025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 15 reason = Timeout
112122025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
112132025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
112142025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
112152025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
112162025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
112172025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
112182025-02-28T12:13:10.453ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5cefabf1-6831-46ff-98eb-1a3b5abad8b9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1625, tv_nsec: 91383258 } })
112192025-02-28T12:13:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
112202025-02-28T12:13:10.453ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5ebff2d9-48b6-4fe6-b852-684bb14b4955.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bddfb0f9-1d7b-45e4-b3a5-35aa6792956b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1625, tv_nsec: 91505431 } })
112212025-02-28T12:13:10.454ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
112222025-02-28T12:13:10.454ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.188489ms iteration = 15 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
112232025-02-28T12:13:10.454ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
112242025-02-28T12:13:10.454ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.284641ms iteration = 15 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
112252025-02-28T12:13:10.454ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
112262025-02-28T12:13:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
112272025-02-28T12:13:10.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 85 reason = Timeout
112282025-02-28T12:13:10.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 16 reason = Timeout
112292025-02-28T12:13:10.455ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
112302025-02-28T12:13:10.455ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 29 reason = Timeout
112312025-02-28T12:13:10.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
112322025-02-28T12:13:10.455ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 9.74\u{b5}s iteration = 16 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
112332025-02-28T12:13:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
112342025-02-28T12:13:10.455ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-28 12:03:10.452823850 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
112352025-02-28T12:13:10.455ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 2.849739ms iteration = 15 status = {"expiration":"2025-02-28T12:03:10.452823850Z","pruned":[]}
112362025-02-28T12:13:10.456ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
112372025-02-28T12:13:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
112382025-02-28T12:13:10.456ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1625, tv_nsec: 94498874 } })
112392025-02-28T12:13:10.456ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 15 reason = Timeout
112402025-02-28T12:13:10.456ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 29 reason = Timeout
112412025-02-28T12:13:10.456ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
112422025-02-28T12:13:10.456ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
112432025-02-28T12:13:10.456ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 29 reason = Timeout
112442025-02-28T12:13:10.456ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
112452025-02-28T12:13:10.456ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 29 reason = Timeout
112462025-02-28T12:13:10.456ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
112472025-02-28T12:13:10.456ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 29 reason = Timeout
112482025-02-28T12:13:10.456ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
112492025-02-28T12:13:10.456ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 29 reason = Timeout
112502025-02-28T12:13:10.456ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 29 reason = Timeout
112512025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
112522025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
112532025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 85 reason = Timeout
112542025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
112552025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
112562025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 29 reason = Timeout
112572025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
112582025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 29 reason = Timeout
112592025-02-28T12:13:10.457ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
112602025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 15 reason = Timeout
112612025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
112622025-02-28T12:13:10.457ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
112632025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
112642025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 31 reason = Timeout
112652025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
112662025-02-28T12:13:10.457ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
112672025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
112682025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
112692025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
112702025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
112712025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
112722025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
112732025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
112742025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
112752025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
112762025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
112772025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
112782025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
112792025-02-28T12:13:10.457ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1625, tv_nsec: 95630631 } })
112802025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 29 reason = Timeout
112812025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
112822025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 29 reason = Timeout
112832025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
112842025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 15 reason = Timeout
112852025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
112862025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 15 reason = Timeout
112872025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
112882025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 29 reason = Timeout
112892025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
112902025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 29 reason = Timeout
112912025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
112922025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 29 reason = Timeout
112932025-02-28T12:13:10.457ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
112942025-02-28T12:13:10.458ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112952025-02-28T12:13:10.458ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
112962025-02-28T12:13:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
112972025-02-28T12:13:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
112982025-02-28T12:13:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
112992025-02-28T12:13:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
113002025-02-28T12:13:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
113012025-02-28T12:13:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
113022025-02-28T12:13:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
113032025-02-28T12:13:10.458ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113042025-02-28T12:13:10.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113052025-02-28T12:13:10.459ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113062025-02-28T12:13:10.460ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
113072025-02-28T12:13:10.460ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
113082025-02-28T12:13:10.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
113092025-02-28T12:13:10.460ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.667827ms iteration = 29 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
113102025-02-28T12:13:10.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
113112025-02-28T12:13:10.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
113122025-02-28T12:13:10.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
113132025-02-28T12:13:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
113142025-02-28T12:13:10.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
113152025-02-28T12:13:10.465ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113162025-02-28T12:13:10.465ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113172025-02-28T12:13:10.468ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113182025-02-28T12:13:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
113192025-02-28T12:13:10.468ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
113202025-02-28T12:13:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
113212025-02-28T12:13:10.468ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
113222025-02-28T12:13:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
113232025-02-28T12:13:10.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
113242025-02-28T12:13:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
113252025-02-28T12:13:10.470ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
113262025-02-28T12:13:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
113272025-02-28T12:13:10.473ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113282025-02-28T12:13:10.474ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113292025-02-28T12:13:10.475ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
113302025-02-28T12:13:10.475ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
113312025-02-28T12:13:10.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
113322025-02-28T12:13:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
113332025-02-28T12:13:10.476ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
113342025-02-28T12:13:10.476ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
113352025-02-28T12:13:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
113362025-02-28T12:13:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
113372025-02-28T12:13:10.478ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 20.934575ms iteration = 15 status = {"errors":[],"found_port_ok":[]}
113382025-02-28T12:13:10.480ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
113392025-02-28T12:13:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
113402025-02-28T12:13:10.481ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
113412025-02-28T12:13:10.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
113422025-02-28T12:13:10.482ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
113432025-02-28T12:13:10.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
113442025-02-28T12:13:10.483ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1625, tv_nsec: 120959472 } })
113452025-02-28T12:13:10.483ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
113462025-02-28T12:13:10.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
113472025-02-28T12:13:10.483ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
113482025-02-28T12:13:10.485ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113492025-02-28T12:13:10.489ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
113502025-02-28T12:13:10.489ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
113512025-02-28T12:13:10.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
113522025-02-28T12:13:10.494ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113532025-02-28T12:13:10.497ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
113542025-02-28T12:13:10.497ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
113552025-02-28T12:13:10.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
113562025-02-28T12:13:10.506ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
113572025-02-28T12:13:10.506ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
113582025-02-28T12:13:10.506ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
113592025-02-28T12:13:10.506ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
113602025-02-28T12:13:10.506ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
113612025-02-28T12:13:10.506ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
113622025-02-28T12:13:10.506ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
113632025-02-28T12:13:10.506ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
113642025-02-28T12:13:10.506ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
113652025-02-28T12:13:10.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
113662025-02-28T12:13:10.509ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
113672025-02-28T12:13:10.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
113682025-02-28T12:13:10.515ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 57.95468ms iteration = 29 status = {"errors":[],"garbage_collect_requested":[]}
113692025-02-28T12:13:10.516ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
113702025-02-28T12:13:10.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
113712025-02-28T12:13:10.518ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113722025-02-28T12:13:10.521ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
113732025-02-28T12:13:10.521ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
113742025-02-28T12:13:10.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
113752025-02-28T12:13:10.521ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
113762025-02-28T12:13:10.521ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
113772025-02-28T12:13:10.521ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
113782025-02-28T12:13:10.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
113792025-02-28T12:13:10.525ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 68.025192ms iteration = 85 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
113802025-02-28T12:13:10.526ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113812025-02-28T12:13:10.529ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
113822025-02-28T12:13:10.529ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
113832025-02-28T12:13:10.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
113842025-02-28T12:13:10.529ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
113852025-02-28T12:13:10.529ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
113862025-02-28T12:13:10.529ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 71.997258ms iteration = 15 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
113872025-02-28T12:13:10.531ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
113882025-02-28T12:13:10.531ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113892025-02-28T12:13:10.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
113902025-02-28T12:13:10.535ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
113912025-02-28T12:13:10.535ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
113922025-02-28T12:13:10.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
113932025-02-28T12:13:10.535ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 77.097831ms iteration = 29 status = {"errors":[],"finish_invoked_ok":[]}
113942025-02-28T12:13:10.535ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
113952025-02-28T12:13:10.535ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
113962025-02-28T12:13:10.535ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
113972025-02-28T12:13:10.535ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
113982025-02-28T12:13:10.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
113992025-02-28T12:13:10.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
114002025-02-28T12:13:10.538ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 80.541084ms iteration = 29 status = {"errors":[],"requests_created_ok":[]}
114012025-02-28T12:13:10.538ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
114022025-02-28T12:13:10.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
114032025-02-28T12:13:10.539ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
114042025-02-28T12:13:10.539ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 84.49976ms iteration = 85 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
114052025-02-28T12:13:10.542ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
114062025-02-28T12:13:10.542ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
114072025-02-28T12:13:10.542ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 85.581016ms iteration = 15 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
114082025-02-28T12:13:10.543ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
114092025-02-28T12:13:10.543ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 86.338864ms iteration = 29 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
114102025-02-28T12:13:10.546ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
114112025-02-28T12:13:10.546ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
114122025-02-28T12:13:10.549ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
114132025-02-28T12:13:10.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
114142025-02-28T12:13:10.555ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 97.610006ms iteration = 29 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
114152025-02-28T12:13:10.556ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
114162025-02-28T12:13:10.556ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
114172025-02-28T12:13:10.564ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114182025-02-28T12:13:10.564ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
114192025-02-28T12:13:10.565ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
114202025-02-28T12:13:10.567ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
114212025-02-28T12:13:10.567ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
114222025-02-28T12:13:10.567ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
114232025-02-28T12:13:10.569ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
114242025-02-28T12:13:10.569ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 0 version = 2
114252025-02-28T12:13:10.569ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
114262025-02-28T12:13:10.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
114272025-02-28T12:13:10.570ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114282025-02-28T12:13:10.571ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114292025-02-28T12:13:10.573ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
114302025-02-28T12:13:10.573ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
114312025-02-28T12:13:10.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
114322025-02-28T12:13:10.574ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
114332025-02-28T12:13:10.574ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
114342025-02-28T12:13:10.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
114352025-02-28T12:13:10.574ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
114362025-02-28T12:13:10.575ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
114372025-02-28T12:13:10.576ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114382025-02-28T12:13:10.576ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 1 version = 2
114392025-02-28T12:13:10.576ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
114402025-02-28T12:13:10.576ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-02-28 11:59:52.542241 UTC dns_group = external generation = 2
114412025-02-28T12:13:10.576ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 123.803098ms iteration = 15 status = {"generation":2}
114422025-02-28T12:13:10.576ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114432025-02-28T12:13:10.577ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 120.678763ms iteration = 29 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
114442025-02-28T12:13:10.578ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 120.96618ms iteration = 29 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
114452025-02-28T12:13:10.579ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
114462025-02-28T12:13:10.579ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
114472025-02-28T12:13:10.579ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
114482025-02-28T12:13:10.579ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
114492025-02-28T12:13:10.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
114502025-02-28T12:13:10.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
114512025-02-28T12:13:10.580ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
114522025-02-28T12:13:10.580ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
114532025-02-28T12:13:10.580ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 123.216194ms iteration = 29 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
114542025-02-28T12:13:10.581ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
114552025-02-28T12:13:10.581ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 124.573768ms iteration = 29 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
114562025-02-28T12:13:10.585ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114572025-02-28T12:13:10.588ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
114582025-02-28T12:13:10.588ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
114592025-02-28T12:13:10.588ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
114602025-02-28T12:13:10.590ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 134.006165ms iteration = 29 status = 0
114612025-02-28T12:13:10.591ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
114622025-02-28T12:13:10.591ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
114632025-02-28T12:13:10.591ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
114642025-02-28T12:13:10.592ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "88bdf527-8479-435d-980a-3620c72c5bad", "content-length": "26", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} })
114652025-02-28T12:13:10.592ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
114662025-02-28T12:13:10.594ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
114672025-02-28T12:13:10.594ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
114682025-02-28T12:13:10.601ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114692025-02-28T12:13:10.604ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
114702025-02-28T12:13:10.604ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
114712025-02-28T12:13:10.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
114722025-02-28T12:13:10.606ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
114732025-02-28T12:13:10.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
114742025-02-28T12:13:10.611ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114752025-02-28T12:13:10.614ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
114762025-02-28T12:13:10.614ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
114772025-02-28T12:13:10.614ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
114782025-02-28T12:13:10.620ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
114792025-02-28T12:13:10.620ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
114802025-02-28T12:13:10.620ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d16839a5-be8c-4ff5-9e25-fd0a54eeefea", "content-length": "26", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} })
114812025-02-28T12:13:10.620ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
114822025-02-28T12:13:10.623ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
114832025-02-28T12:13:10.650ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
114842025-02-28T12:13:10.650ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
114852025-02-28T12:13:10.650ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e68bda31-aab0-4b5c-8643-61d1d07794ea", "content-length": "26", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} })
114862025-02-28T12:13:10.650ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
114872025-02-28T12:13:10.652ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
114882025-02-28T12:13:10.654ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
114892025-02-28T12:13:10.654ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
114902025-02-28T12:13:10.654ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
114912025-02-28T12:13:10.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
114922025-02-28T12:13:10.655ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114932025-02-28T12:13:10.657ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
114942025-02-28T12:13:10.657ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 200.757657ms iteration = 15 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
114952025-02-28T12:13:10.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
114962025-02-28T12:13:10.660ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
114972025-02-28T12:13:10.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
114982025-02-28T12:13:10.676ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-28T11:59:08.719399Z, time_modified: 2025-02-28T11:59:08.719399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }) }
114992025-02-28T12:13:10.676ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 222.995173ms iteration = 15 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]}},"default_endpoint":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
115002025-02-28T12:13:10.688ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115012025-02-28T12:13:10.692ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
115022025-02-28T12:13:10.692ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
115032025-02-28T12:13:10.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
115042025-02-28T12:13:10.771ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 314.512073ms iteration = 31 status = {"physical_disks_added":0}
115052025-02-28T12:13:10.812ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 359.309904ms iteration = 29 status = {}
115062025-02-28T12:13:10.813ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
115072025-02-28T12:13:10.813ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "edd70d5e-0a16-4cca-bed8-edac2a8cf482", "content-length": "2", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} })
115082025-02-28T12:13:10.817ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
115092025-02-28T12:13:10.818ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
115102025-02-28T12:13:10.818ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
115112025-02-28T12:13:10.822ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
115122025-02-28T12:13:10.822ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "08bd8fbc-26bc-437b-944b-eb4ea6dcebda", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
115132025-02-28T12:13:10.822ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
115142025-02-28T12:13:10.824ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "f002333b-3357-4717-90e5-d65f6035c155", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
115152025-02-28T12:13:10.824ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
115162025-02-28T12:13:10.826ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
115172025-02-28T12:13:10.826ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 373.527077ms iteration = 17 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
115182025-02-28T12:13:10.833ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
115192025-02-28T12:13:10.833ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
115202025-02-28T12:13:10.833ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "8144679e-df40-4931-bb1d-ec45fe9510cf", "content-length": "372", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} })
115212025-02-28T12:13:10.833ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
115222025-02-28T12:13:10.834ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
115232025-02-28T12:13:10.836ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
115242025-02-28T12:13:10.836ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
115252025-02-28T12:13:10.836ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-02-28 11:59:51.915187 UTC dns_group = internal generation = 1
115262025-02-28T12:13:10.836ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 383.786735ms iteration = 15 status = {"generation":1}
115272025-02-28T12:13:10.836ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 384.078792ms iteration = 29 status = null
115282025-02-28T12:13:10.886ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
115292025-02-28T12:13:10.886ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
115302025-02-28T12:13:10.886ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "28247cae-9576-48a4-a906-efaf9c7606b2", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} })
115312025-02-28T12:13:10.886ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
115322025-02-28T12:13:10.887ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
115332025-02-28T12:13:10.889ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
115342025-02-28T12:13:10.889ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
115352025-02-28T12:13:10.892ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
115362025-02-28T12:13:10.893ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
115372025-02-28T12:13:10.900ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
115382025-02-28T12:13:10.901ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "e884b8d8-65e6-4d3b-b647-35626f07a56b", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
115392025-02-28T12:13:10.901ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
115402025-02-28T12:13:10.902ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "e9125b7c-3b9d-4246-9233-4aca3d65e70a", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
115412025-02-28T12:13:10.902ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
115422025-02-28T12:13:10.904ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "8b5d2dda-6313-4cd3-8683-1e76eeb34513", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
115432025-02-28T12:13:10.904ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
115442025-02-28T12:13:10.905ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
115452025-02-28T12:13:10.905ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 453.031656ms iteration = 17 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
115462025-02-28T12:13:10.913ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
115472025-02-28T12:13:10.916ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "96bbe319-6900-44c6-a1f7-b39e4606c499", "content-length": "691", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} })
115482025-02-28T12:13:10.916ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
115492025-02-28T12:13:10.916ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
115502025-02-28T12:13:10.916ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
115512025-02-28T12:13:10.917ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
115522025-02-28T12:13:10.917ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
115532025-02-28T12:13:10.917ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "0c2b3ea7-d721-492b-a0f8-d1ebffb1fe26", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} })
115542025-02-28T12:13:10.918ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 461.654954ms iteration = 29 status = {}
115552025-02-28T12:13:10.920ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 463.084779ms iteration = 29 status = {}
115562025-02-28T12:13:10.961ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
115572025-02-28T12:13:10.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
115582025-02-28T12:13:10.963ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
115592025-02-28T12:13:10.963ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
115602025-02-28T12:13:10.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
115612025-02-28T12:13:10.974ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
115622025-02-28T12:13:10.974ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b70262c-e2eb-4a99-83f5-b19acb28a7bc", "content-length": "134", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} })
115632025-02-28T12:13:10.974ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
115642025-02-28T12:13:10.974ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
115652025-02-28T12:13:10.974ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
115662025-02-28T12:13:10.974ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
115672025-02-28T12:13:10.974ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
115682025-02-28T12:13:10.974ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
115692025-02-28T12:13:10.974ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
115702025-02-28T12:13:10.974ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
115712025-02-28T12:13:10.975ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
115722025-02-28T12:13:10.975ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
115732025-02-28T12:13:10.975ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
115742025-02-28T12:13:10.975ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
115752025-02-28T12:13:10.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
115762025-02-28T12:13:10.977ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
115772025-02-28T12:13:10.977ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "54fd0441-1c39-4f10-adaf-3e03ed2e7eaa", "content-length": "2", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} })
115782025-02-28T12:13:10.977ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
115792025-02-28T12:13:10.977ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "475be355-0858-495e-ba54-312f93c4f475", "content-length": "51", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} })
115802025-02-28T12:13:10.977ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
115812025-02-28T12:13:10.977ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
115822025-02-28T12:13:11.078ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "1a5fe9b4-6bcd-44a1-b2c4-f5b2c86a34d1", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} })
115832025-02-28T12:13:11.078ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
115842025-02-28T12:13:11.078ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
115852025-02-28T12:13:11.078ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "e675393b-50f0-423f-9939-83e5bab5dd22", "content-length": "682", "date": "Fri, 28 Feb 2025 12:13:10 GMT"} })
115862025-02-28T12:13:11.078ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
115872025-02-28T12:13:11.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
115882025-02-28T12:13:11.082ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
115892025-02-28T12:13:11.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
115902025-02-28T12:13:11.086ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
115912025-02-28T12:13:11.086ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
115922025-02-28T12:13:11.088ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
115932025-02-28T12:13:11.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
115942025-02-28T12:13:11.095ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 638.74498ms iteration = 29 status = {}
115952025-02-28T12:13:14.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 297 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = c443ca82-f92b-4596-b0a3-3f0980ec79e9 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
115962025-02-28T12:13:20.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 86 reason = Timeout
115972025-02-28T12:13:20.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
115982025-02-28T12:13:20.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
115992025-02-28T12:13:20.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 86 reason = Timeout
116002025-02-28T12:13:20.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
116012025-02-28T12:13:20.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116022025-02-28T12:13:20.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
116032025-02-28T12:13:20.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
116042025-02-28T12:13:20.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
116052025-02-28T12:13:20.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
116062025-02-28T12:13:20.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
116072025-02-28T12:13:20.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
116082025-02-28T12:13:20.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116092025-02-28T12:13:20.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
116102025-02-28T12:13:20.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
116112025-02-28T12:13:20.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
116122025-02-28T12:13:20.480ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
116132025-02-28T12:13:20.480ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.88965ms iteration = 86 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
116142025-02-28T12:13:20.488ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
116152025-02-28T12:13:20.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
116162025-02-28T12:13:20.519ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.228752ms iteration = 86 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
116172025-02-28T12:13:24.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 268 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = cc4ac9db-7a82-4b01-8c8b-9cb909f74980 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
116182025-02-28T12:13:30.454ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 87 reason = Timeout
116192025-02-28T12:13:30.454ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
116202025-02-28T12:13:30.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
116212025-02-28T12:13:30.456ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 87 reason = Timeout
116222025-02-28T12:13:30.456ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
116232025-02-28T12:13:30.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
116242025-02-28T12:13:30.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116252025-02-28T12:13:30.459ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
116262025-02-28T12:13:30.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
116272025-02-28T12:13:30.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
116282025-02-28T12:13:30.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
116292025-02-28T12:13:30.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
116302025-02-28T12:13:30.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116312025-02-28T12:13:30.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
116322025-02-28T12:13:30.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
116332025-02-28T12:13:30.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
116342025-02-28T12:13:30.482ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
116352025-02-28T12:13:30.482ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 27.199246ms iteration = 87 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
116362025-02-28T12:13:30.491ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
116372025-02-28T12:13:30.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
116382025-02-28T12:13:30.493ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 36.390378ms iteration = 87 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
116392025-02-28T12:13:34.260ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 416 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 592c20d3-7c58-44e9-8cb9-7bc259e1514f response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
116402025-02-28T12:13:40.452ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 30 reason = Timeout
116412025-02-28T12:13:40.452ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 30 reason = Timeout
116422025-02-28T12:13:40.452ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
116432025-02-28T12:13:40.452ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
116442025-02-28T12:13:40.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
116452025-02-28T12:13:40.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
116462025-02-28T12:13:40.454ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 88 reason = Timeout
116472025-02-28T12:13:40.454ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 30 reason = Timeout
116482025-02-28T12:13:40.454ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
116492025-02-28T12:13:40.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
116502025-02-28T12:13:40.455ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1655, tv_nsec: 93522780 } })
116512025-02-28T12:13:40.455ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1655, tv_nsec: 93652663 } })
116522025-02-28T12:13:40.455ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
116532025-02-28T12:13:40.456ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
116542025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 88 reason = Timeout
116552025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 30 reason = Timeout
116562025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
116572025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
116582025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 32 reason = Timeout
116592025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
116602025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 30 reason = Timeout
116612025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
116622025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
116632025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 30 reason = Timeout
116642025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
116652025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 30 reason = Timeout
116662025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 30 reason = Timeout
116672025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
116682025-02-28T12:13:40.457ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
116692025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
116702025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 30 reason = Timeout
116712025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 30 reason = Timeout
116722025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
116732025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 30 reason = Timeout
116742025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
116752025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
116762025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116772025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
116782025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
116792025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
116802025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
116812025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
116822025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
116832025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
116842025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
116852025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
116862025-02-28T12:13:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
116872025-02-28T12:13:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 30 reason = Timeout
116882025-02-28T12:13:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 30 reason = Timeout
116892025-02-28T12:13:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
116902025-02-28T12:13:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
116912025-02-28T12:13:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 30 reason = Timeout
116922025-02-28T12:13:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
116932025-02-28T12:13:40.458ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 30 reason = Timeout
116942025-02-28T12:13:40.458ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
116952025-02-28T12:13:40.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 30 reason = Timeout
116962025-02-28T12:13:40.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
116972025-02-28T12:13:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
116982025-02-28T12:13:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
116992025-02-28T12:13:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
117002025-02-28T12:13:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
117012025-02-28T12:13:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
117022025-02-28T12:13:40.459ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.182791ms iteration = 30 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
117032025-02-28T12:13:40.459ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117042025-02-28T12:13:40.459ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117052025-02-28T12:13:40.460ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117062025-02-28T12:13:40.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
117072025-02-28T12:13:40.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
117082025-02-28T12:13:40.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
117092025-02-28T12:13:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
117102025-02-28T12:13:40.462ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
117112025-02-28T12:13:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
117122025-02-28T12:13:40.462ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
117132025-02-28T12:13:40.462ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
117142025-02-28T12:13:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
117152025-02-28T12:13:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
117162025-02-28T12:13:40.463ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
117172025-02-28T12:13:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
117182025-02-28T12:13:40.464ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117192025-02-28T12:13:40.466ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117202025-02-28T12:13:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
117212025-02-28T12:13:40.467ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
117222025-02-28T12:13:40.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
117232025-02-28T12:13:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
117242025-02-28T12:13:40.468ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
117252025-02-28T12:13:40.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
117262025-02-28T12:13:40.472ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
117272025-02-28T12:13:40.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
117282025-02-28T12:13:40.473ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
117292025-02-28T12:13:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
117302025-02-28T12:13:40.473ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
117312025-02-28T12:13:40.473ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1655, tv_nsec: 111527905 } })
117322025-02-28T12:13:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
117332025-02-28T12:13:40.474ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
117342025-02-28T12:13:40.509ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
117352025-02-28T12:13:40.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
117362025-02-28T12:13:40.509ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
117372025-02-28T12:13:40.509ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
117382025-02-28T12:13:40.509ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
117392025-02-28T12:13:40.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
117402025-02-28T12:13:40.514ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117412025-02-28T12:13:40.515ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
117422025-02-28T12:13:40.515ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
117432025-02-28T12:13:40.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
117442025-02-28T12:13:40.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
117452025-02-28T12:13:40.518ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
117462025-02-28T12:13:40.518ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
117472025-02-28T12:13:40.518ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117482025-02-28T12:13:40.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
117492025-02-28T12:13:40.518ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 59.751642ms iteration = 30 status = {"errors":[],"requests_created_ok":[]}
117502025-02-28T12:13:40.519ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
117512025-02-28T12:13:40.519ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
117522025-02-28T12:13:40.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
117532025-02-28T12:13:40.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
117542025-02-28T12:13:40.520ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 61.430573ms iteration = 30 status = {"errors":[],"finish_invoked_ok":[]}
117552025-02-28T12:13:40.520ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
117562025-02-28T12:13:40.520ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
117572025-02-28T12:13:40.520ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
117582025-02-28T12:13:40.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
117592025-02-28T12:13:40.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
117602025-02-28T12:13:40.526ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
117612025-02-28T12:13:40.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
117622025-02-28T12:13:40.529ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 70.972314ms iteration = 30 status = {"errors":[],"garbage_collect_requested":[]}
117632025-02-28T12:13:40.530ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
117642025-02-28T12:13:40.530ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
117652025-02-28T12:13:40.531ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9bb0be6-1cc9-437a-967a-a1af2c5881db", "content-length": "26", "date": "Fri, 28 Feb 2025 12:13:40 GMT"} })
117662025-02-28T12:13:40.531ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
117672025-02-28T12:13:40.531ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117682025-02-28T12:13:40.532ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
117692025-02-28T12:13:40.534ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
117702025-02-28T12:13:40.534ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
117712025-02-28T12:13:40.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
117722025-02-28T12:13:40.535ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117732025-02-28T12:13:40.538ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
117742025-02-28T12:13:40.538ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
117752025-02-28T12:13:40.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
117762025-02-28T12:13:40.555ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 98.18625ms iteration = 30 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
117772025-02-28T12:13:40.555ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
117782025-02-28T12:13:40.555ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 100.722071ms iteration = 88 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
117792025-02-28T12:13:40.557ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
117802025-02-28T12:13:40.557ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
117812025-02-28T12:13:40.557ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
117822025-02-28T12:13:40.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
117832025-02-28T12:13:40.559ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
117842025-02-28T12:13:40.559ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
117852025-02-28T12:13:40.559ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d28dbdf-fbae-4626-a32f-8b2b6ccb6227", "content-length": "26", "date": "Fri, 28 Feb 2025 12:13:40 GMT"} })
117862025-02-28T12:13:40.559ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
117872025-02-28T12:13:40.561ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117882025-02-28T12:13:40.561ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
117892025-02-28T12:13:40.562ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117902025-02-28T12:13:40.562ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 104.103563ms iteration = 30 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
117912025-02-28T12:13:40.564ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
117922025-02-28T12:13:40.564ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
117932025-02-28T12:13:40.564ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
117942025-02-28T12:13:40.565ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
117952025-02-28T12:13:40.565ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
117962025-02-28T12:13:40.565ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
117972025-02-28T12:13:40.565ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
117982025-02-28T12:13:40.565ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 108.250083ms iteration = 30 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
117992025-02-28T12:13:40.565ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 108.369496ms iteration = 30 status = 0
118002025-02-28T12:13:40.567ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
118012025-02-28T12:13:40.567ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 109.895843ms iteration = 30 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
118022025-02-28T12:13:40.570ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
118032025-02-28T12:13:40.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
118042025-02-28T12:13:40.574ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 117.211339ms iteration = 88 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
118052025-02-28T12:13:40.601ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
118062025-02-28T12:13:40.601ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
118072025-02-28T12:13:40.601ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 143.976046ms iteration = 30 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
118082025-02-28T12:13:40.606ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
118092025-02-28T12:13:40.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
118102025-02-28T12:13:40.622ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 164.147732ms iteration = 30 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
118112025-02-28T12:13:40.641ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
118122025-02-28T12:13:40.641ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
118132025-02-28T12:13:40.642ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "da662798-5f0f-4c99-b690-8ee6ab2b836c", "content-length": "26", "date": "Fri, 28 Feb 2025 12:13:40 GMT"} })
118142025-02-28T12:13:40.642ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
118152025-02-28T12:13:40.645ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
118162025-02-28T12:13:40.656ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 199.662039ms iteration = 32 status = {"physical_disks_added":0}
118172025-02-28T12:13:40.737ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
118182025-02-28T12:13:40.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
118192025-02-28T12:13:40.760ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
118202025-02-28T12:13:40.760ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
118212025-02-28T12:13:40.760ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0c9d8db-0772-400e-a452-e4d3232ecee5", "content-length": "372", "date": "Fri, 28 Feb 2025 12:13:40 GMT"} })
118222025-02-28T12:13:40.760ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
118232025-02-28T12:13:40.760ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
118242025-02-28T12:13:40.774ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118252025-02-28T12:13:40.777ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
118262025-02-28T12:13:40.777ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
118272025-02-28T12:13:40.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
118282025-02-28T12:13:40.778ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118292025-02-28T12:13:40.781ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
118302025-02-28T12:13:40.781ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
118312025-02-28T12:13:40.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
118322025-02-28T12:13:40.781ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
118332025-02-28T12:13:40.781ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
118342025-02-28T12:13:40.781ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "033d6ade-5c82-4bc1-a6a8-d771280f1968", "date": "Fri, 28 Feb 2025 12:13:40 GMT"} })
118352025-02-28T12:13:40.781ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
118362025-02-28T12:13:40.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
118372025-02-28T12:13:40.784ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
118382025-02-28T12:13:40.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
118392025-02-28T12:13:40.788ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
118402025-02-28T12:13:40.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
118412025-02-28T12:13:40.790ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
118422025-02-28T12:13:40.792ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "40492bb4-6701-4522-893a-953a0c459a0c", "content-length": "2", "date": "Fri, 28 Feb 2025 12:13:40 GMT"} })
118432025-02-28T12:13:40.796ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
118442025-02-28T12:13:40.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
118452025-02-28T12:13:40.799ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 346.763981ms iteration = 30 status = null
118462025-02-28T12:13:40.801ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
118472025-02-28T12:13:40.802ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
118482025-02-28T12:13:40.802ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "d8f1d9ce-a91e-4968-a066-9c0e61faf5e6", "date": "Fri, 28 Feb 2025 12:13:40 GMT"} })
118492025-02-28T12:13:40.804ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 347.430677ms iteration = 30 status = {}
118502025-02-28T12:13:40.805ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 352.94171ms iteration = 30 status = {}
118512025-02-28T12:13:40.884ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
118522025-02-28T12:13:40.885ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
118532025-02-28T12:13:40.887ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
118542025-02-28T12:13:40.887ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
118552025-02-28T12:13:40.887ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
118562025-02-28T12:13:40.898ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
118572025-02-28T12:13:40.898ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1658ff6-8ac7-4ff7-ab42-94b8e9dae90e", "content-length": "134", "date": "Fri, 28 Feb 2025 12:13:40 GMT"} })
118582025-02-28T12:13:40.898ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
118592025-02-28T12:13:40.898ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
118602025-02-28T12:13:40.899ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
118612025-02-28T12:13:40.899ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
118622025-02-28T12:13:40.899ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
118632025-02-28T12:13:40.899ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
118642025-02-28T12:13:40.899ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
118652025-02-28T12:13:40.899ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
118662025-02-28T12:13:40.899ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
118672025-02-28T12:13:40.899ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
118682025-02-28T12:13:40.899ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
118692025-02-28T12:13:40.899ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
118702025-02-28T12:13:40.899ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
118712025-02-28T12:13:40.901ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
118722025-02-28T12:13:40.901ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "311ff14f-3e3d-43f7-94d1-f8057f1c81f6", "content-length": "2", "date": "Fri, 28 Feb 2025 12:13:40 GMT"} })
118732025-02-28T12:13:40.901ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
118742025-02-28T12:13:40.901ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "827022c0-e57b-4f5c-a4e7-fa16d02220f8", "content-length": "51", "date": "Fri, 28 Feb 2025 12:13:40 GMT"} })
118752025-02-28T12:13:40.901ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
118762025-02-28T12:13:40.901ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
118772025-02-28T12:13:40.905ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
118782025-02-28T12:13:40.988ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "792b76e8-f0e3-41b3-938e-2d2c858a575f", "date": "Fri, 28 Feb 2025 12:13:40 GMT"} })
118792025-02-28T12:13:40.988ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
118802025-02-28T12:13:40.988ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
118812025-02-28T12:13:40.989ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fc52710-f091-4017-8a89-021a7639759b", "content-length": "682", "date": "Fri, 28 Feb 2025 12:13:40 GMT"} })
118822025-02-28T12:13:40.989ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
118832025-02-28T12:13:40.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
118842025-02-28T12:13:40.989ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "617b157d-a255-4752-8cfd-15a191ee779e", "content-length": "691", "date": "Fri, 28 Feb 2025 12:13:40 GMT"} })
118852025-02-28T12:13:40.989ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
118862025-02-28T12:13:40.989ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
118872025-02-28T12:13:40.989ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
118882025-02-28T12:13:40.990ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 533.583941ms iteration = 30 status = {}
118892025-02-28T12:13:40.993ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
118902025-02-28T12:13:40.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
118912025-02-28T12:13:40.997ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
118922025-02-28T12:13:40.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
118932025-02-28T12:13:41.001ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
118942025-02-28T12:13:41.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
118952025-02-28T12:13:41.011ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 554.158817ms iteration = 30 status = {}
118962025-02-28T12:13:44.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 297 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 4ea3d878-b7af-4ed1-b0f9-be51488f0e71 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
118972025-02-28T12:13:50.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 89 reason = Timeout
118982025-02-28T12:13:50.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
118992025-02-28T12:13:50.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
119002025-02-28T12:13:50.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 89 reason = Timeout
119012025-02-28T12:13:50.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
119022025-02-28T12:13:50.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
119032025-02-28T12:13:50.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119042025-02-28T12:13:50.459ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
119052025-02-28T12:13:50.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
119062025-02-28T12:13:50.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
119072025-02-28T12:13:50.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
119082025-02-28T12:13:50.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
119092025-02-28T12:13:50.487ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
119102025-02-28T12:13:50.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
119112025-02-28T12:13:50.492ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119122025-02-28T12:13:50.495ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
119132025-02-28T12:13:50.495ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
119142025-02-28T12:13:50.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
119152025-02-28T12:13:50.514ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
119162025-02-28T12:13:50.514ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 59.65899ms iteration = 89 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
119172025-02-28T12:13:50.519ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 61.999737ms iteration = 89 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
119182025-02-28T12:13:54.252ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::d]:42136
119192025-02-28T12:13:54.252ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42136 req_id = c122dba4-77e1-4e3e-9b15-aef828442e42 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
119202025-02-28T12:13:54.253ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42136 req_id = c122dba4-77e1-4e3e-9b15-aef828442e42 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
119212025-02-28T12:13:54.255ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3073 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42136 req_id = c122dba4-77e1-4e3e-9b15-aef828442e42 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
119222025-02-28T12:13:54.256ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42136 req_id = 61c11b41-a770-4dd4-bda0-307620040f96 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
119232025-02-28T12:13:54.256ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42136 req_id = 61c11b41-a770-4dd4-bda0-307620040f96 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
119242025-02-28T12:13:54.258ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2477 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42136 req_id = 61c11b41-a770-4dd4-bda0-307620040f96 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
119252025-02-28T12:13:54.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 226 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = 51b1d7d9-6a5a-4039-830a-1edfc5d58f50 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
119262025-02-28T12:14:00.454ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 90 reason = Timeout
119272025-02-28T12:14:00.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
119282025-02-28T12:14:00.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
119292025-02-28T12:14:00.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 90 reason = Timeout
119302025-02-28T12:14:00.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
119312025-02-28T12:14:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
119322025-02-28T12:14:00.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119332025-02-28T12:14:00.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
119342025-02-28T12:14:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
119352025-02-28T12:14:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
119362025-02-28T12:14:00.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
119372025-02-28T12:14:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
119382025-02-28T12:14:00.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
119392025-02-28T12:14:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
119402025-02-28T12:14:00.494ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 37.767992ms iteration = 90 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
119412025-02-28T12:14:00.499ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119422025-02-28T12:14:00.502ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
119432025-02-28T12:14:00.502ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
119442025-02-28T12:14:00.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
119452025-02-28T12:14:00.518ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
119462025-02-28T12:14:00.518ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 63.903302ms iteration = 90 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
119472025-02-28T12:14:04.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 264 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = b86ed011-bf89-42fb-82fa-c920eb98ba1e response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
119482025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 16 reason = Timeout
119492025-02-28T12:14:10.452ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-28 12:04:10.452333783 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
119502025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
119512025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 16 reason = Timeout
119522025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
119532025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 31 reason = Timeout
119542025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
119552025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
119562025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 31 reason = Timeout
119572025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
119582025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 18 reason = Timeout
119592025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 16 reason = Timeout
119602025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
119612025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 16 reason = Timeout
119622025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
119632025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
119642025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 18 reason = Timeout
119652025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 16 reason = Timeout
119662025-02-28T12:14:10.452ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 16 reason = Timeout
119672025-02-28T12:14:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
119682025-02-28T12:14:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
119692025-02-28T12:14:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
119702025-02-28T12:14:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
119712025-02-28T12:14:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
119722025-02-28T12:14:10.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
119732025-02-28T12:14:10.453ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5ebff2d9-48b6-4fe6-b852-684bb14b4955.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bddfb0f9-1d7b-45e4-b3a5-35aa6792956b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2f7ea70-d79d-4652-8315-4cb162fd2ecf.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1685, tv_nsec: 91373948 } })
119742025-02-28T12:14:10.453ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5cefabf1-6831-46ff-98eb-1a3b5abad8b9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("db281835-e35c-480d-b1a2-07f321816d55.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1685, tv_nsec: 91520072 } })
119752025-02-28T12:14:10.453ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
119762025-02-28T12:14:10.453ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.379773ms iteration = 16 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
119772025-02-28T12:14:10.453ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
119782025-02-28T12:14:10.453ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.428594ms iteration = 16 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
119792025-02-28T12:14:10.454ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 91 reason = Timeout
119802025-02-28T12:14:10.454ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 17 reason = Timeout
119812025-02-28T12:14:10.454ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
119822025-02-28T12:14:10.454ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 7a676565-49af-418f-9d1f-e6c070e71120
119832025-02-28T12:14:10.454ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 11.15\u{b5}s iteration = 17 status = {"enabled":false,"target_id":"7a676565-49af-418f-9d1f-e6c070e71120"}
119842025-02-28T12:14:10.454ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 31 reason = Timeout
119852025-02-28T12:14:10.454ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 7 reason = Timeout
119862025-02-28T12:14:10.454ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
119872025-02-28T12:14:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
119882025-02-28T12:14:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
119892025-02-28T12:14:10.455ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
119902025-02-28T12:14:10.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
119912025-02-28T12:14:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
119922025-02-28T12:14:10.455ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
119932025-02-28T12:14:10.455ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-28 12:04:10.452333783 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
119942025-02-28T12:14:10.455ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.249088ms iteration = 16 status = {"expiration":"2025-02-28T12:04:10.452333783Z","pruned":[]}
119952025-02-28T12:14:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
119962025-02-28T12:14:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
119972025-02-28T12:14:10.455ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
119982025-02-28T12:14:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
119992025-02-28T12:14:10.455ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
120002025-02-28T12:14:10.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
120012025-02-28T12:14:10.456ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1685, tv_nsec: 94028732 } })
120022025-02-28T12:14:10.456ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
120032025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
120042025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
120052025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 16 reason = Timeout
120062025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
120072025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 31 reason = Timeout
120082025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
120092025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 31 reason = Timeout
120102025-02-28T12:14:10.456ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
120112025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 16 reason = Timeout
120122025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
120132025-02-28T12:14:10.456ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
120142025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
120152025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 31 reason = Timeout
120162025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
120172025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 31 reason = Timeout
120182025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 31 reason = Timeout
120192025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
120202025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
120212025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 33 reason = Timeout
120222025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
120232025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 31 reason = Timeout
120242025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 91 reason = Timeout
120252025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 4 reason = Timeout
120262025-02-28T12:14:10.456ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
120272025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
120282025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
120292025-02-28T12:14:10.456ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
120302025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 6 reason = Timeout
120312025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 31 reason = Timeout
120322025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
120332025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
120342025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 31 reason = Timeout
120352025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
120362025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
120372025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
120382025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
120392025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
120402025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
120412025-02-28T12:14:10.457ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1685, tv_nsec: 95071988 } })
120422025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
120432025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
120442025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
120452025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
120462025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
120472025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
120482025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
120492025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
120502025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
120512025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
120522025-02-28T12:14:10.457ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
120532025-02-28T12:14:10.457ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120542025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 16 reason = Timeout
120552025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
120562025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 31 reason = Timeout
120572025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
120582025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 31 reason = Timeout
120592025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
120602025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 31 reason = Timeout
120612025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
120622025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 16 reason = Timeout
120632025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
120642025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 4 reason = Timeout
120652025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
120662025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 31 reason = Timeout
120672025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
120682025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 31 reason = Timeout
120692025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
120702025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
120712025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
120722025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
120732025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
120742025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
120752025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
120762025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
120772025-02-28T12:14:10.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
120782025-02-28T12:14:10.459ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
120792025-02-28T12:14:10.459ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
120802025-02-28T12:14:10.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
120812025-02-28T12:14:10.459ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.879647ms iteration = 31 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
120822025-02-28T12:14:10.460ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120832025-02-28T12:14:10.461ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120842025-02-28T12:14:10.463ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120852025-02-28T12:14:10.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
120862025-02-28T12:14:10.464ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
120872025-02-28T12:14:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
120882025-02-28T12:14:10.465ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120892025-02-28T12:14:10.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
120902025-02-28T12:14:10.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
120912025-02-28T12:14:10.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
120922025-02-28T12:14:10.467ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
120932025-02-28T12:14:10.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
120942025-02-28T12:14:10.467ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120952025-02-28T12:14:10.468ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120962025-02-28T12:14:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
120972025-02-28T12:14:10.469ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
120982025-02-28T12:14:10.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
120992025-02-28T12:14:10.470ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
121002025-02-28T12:14:10.470ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
121012025-02-28T12:14:10.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
121022025-02-28T12:14:10.471ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121032025-02-28T12:14:10.471ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121042025-02-28T12:14:10.472ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
121052025-02-28T12:14:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
121062025-02-28T12:14:10.472ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
121072025-02-28T12:14:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
121082025-02-28T12:14:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
121092025-02-28T12:14:10.473ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
121102025-02-28T12:14:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
121112025-02-28T12:14:10.474ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
121122025-02-28T12:14:10.474ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
121132025-02-28T12:14:10.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
121142025-02-28T12:14:10.475ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121152025-02-28T12:14:10.477ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121162025-02-28T12:14:10.477ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
121172025-02-28T12:14:10.477ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
121182025-02-28T12:14:10.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
121192025-02-28T12:14:10.478ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
121202025-02-28T12:14:10.478ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
121212025-02-28T12:14:10.478ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
121222025-02-28T12:14:10.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
121232025-02-28T12:14:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
121242025-02-28T12:14:10.480ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
121252025-02-28T12:14:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
121262025-02-28T12:14:10.484ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 25.464185ms iteration = 16 status = {"errors":[],"found_port_ok":[]}
121272025-02-28T12:14:10.484ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121282025-02-28T12:14:10.486ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
121292025-02-28T12:14:10.486ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
121302025-02-28T12:14:10.486ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
121312025-02-28T12:14:10.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
121322025-02-28T12:14:10.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
121332025-02-28T12:14:10.488ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121342025-02-28T12:14:10.489ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121352025-02-28T12:14:10.490ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
121362025-02-28T12:14:10.491ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
121372025-02-28T12:14:10.491ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
121382025-02-28T12:14:10.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
121392025-02-28T12:14:10.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
121402025-02-28T12:14:10.491ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
121412025-02-28T12:14:10.491ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
121422025-02-28T12:14:10.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
121432025-02-28T12:14:10.493ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 37.111946ms iteration = 91 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
121442025-02-28T12:14:10.494ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121452025-02-28T12:14:10.494ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
121462025-02-28T12:14:10.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
121472025-02-28T12:14:10.496ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121482025-02-28T12:14:10.497ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
121492025-02-28T12:14:10.497ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
121502025-02-28T12:14:10.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
121512025-02-28T12:14:10.498ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1685, tv_nsec: 136530198 } })
121522025-02-28T12:14:10.499ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
121532025-02-28T12:14:10.499ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
121542025-02-28T12:14:10.500ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
121552025-02-28T12:14:10.500ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
121562025-02-28T12:14:10.502ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121572025-02-28T12:14:10.502ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
121582025-02-28T12:14:10.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
121592025-02-28T12:14:10.502ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121602025-02-28T12:14:10.502ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
121612025-02-28T12:14:10.503ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
121622025-02-28T12:14:10.503ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
121632025-02-28T12:14:10.503ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
121642025-02-28T12:14:10.503ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
121652025-02-28T12:14:10.503ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
121662025-02-28T12:14:10.504ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
121672025-02-28T12:14:10.504ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
121682025-02-28T12:14:10.505ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
121692025-02-28T12:14:10.505ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
121702025-02-28T12:14:10.505ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
121712025-02-28T12:14:10.507ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
121722025-02-28T12:14:10.507ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
121732025-02-28T12:14:10.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
121742025-02-28T12:14:10.513ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121752025-02-28T12:14:10.514ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
121762025-02-28T12:14:10.514ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121772025-02-28T12:14:10.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
121782025-02-28T12:14:10.515ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
121792025-02-28T12:14:10.515ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
121802025-02-28T12:14:10.515ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
121812025-02-28T12:14:10.515ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
121822025-02-28T12:14:10.515ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121832025-02-28T12:14:10.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
121842025-02-28T12:14:10.517ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
121852025-02-28T12:14:10.517ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
121862025-02-28T12:14:10.517ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
121872025-02-28T12:14:10.517ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
121882025-02-28T12:14:10.517ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
121892025-02-28T12:14:10.517ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
121902025-02-28T12:14:10.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
121912025-02-28T12:14:10.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
121922025-02-28T12:14:10.518ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
121932025-02-28T12:14:10.518ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
121942025-02-28T12:14:10.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
121952025-02-28T12:14:10.519ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
121962025-02-28T12:14:10.519ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
121972025-02-28T12:14:10.519ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121982025-02-28T12:14:10.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
121992025-02-28T12:14:10.522ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
122002025-02-28T12:14:10.522ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
122012025-02-28T12:14:10.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
122022025-02-28T12:14:10.523ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122032025-02-28T12:14:10.523ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
122042025-02-28T12:14:10.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
122052025-02-28T12:14:10.527ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
122062025-02-28T12:14:10.527ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
122072025-02-28T12:14:10.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
122082025-02-28T12:14:10.530ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
122092025-02-28T12:14:10.530ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
122102025-02-28T12:14:10.533ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 74.729653ms iteration = 31 status = {"errors":[],"requests_created_ok":[]}
122112025-02-28T12:14:10.533ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 75.288068ms iteration = 31 status = {"errors":[],"garbage_collect_requested":[]}
122122025-02-28T12:14:10.534ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 75.900612ms iteration = 31 status = {"errors":[],"finish_invoked_ok":[]}
122132025-02-28T12:14:10.534ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
122142025-02-28T12:14:10.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
122152025-02-28T12:14:10.535ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
122162025-02-28T12:14:10.535ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
122172025-02-28T12:14:10.535ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 77.268385ms iteration = 16 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
122182025-02-28T12:14:10.536ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122192025-02-28T12:14:10.536ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 81.824015ms iteration = 7 status = {"nsuccess":5}
122202025-02-28T12:14:10.539ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
122212025-02-28T12:14:10.539ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
122222025-02-28T12:14:10.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
122232025-02-28T12:14:10.541ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
122242025-02-28T12:14:10.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
122252025-02-28T12:14:10.542ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
122262025-02-28T12:14:10.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
122272025-02-28T12:14:10.547ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
122282025-02-28T12:14:10.547ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 92.423121ms iteration = 91 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
122292025-02-28T12:14:10.553ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
122302025-02-28T12:14:10.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
122312025-02-28T12:14:10.558ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
122322025-02-28T12:14:10.558ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
122332025-02-28T12:14:10.559ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
122342025-02-28T12:14:10.560ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
122352025-02-28T12:14:10.561ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
122362025-02-28T12:14:10.562ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
122372025-02-28T12:14:10.562ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
122382025-02-28T12:14:10.562ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
122392025-02-28T12:14:10.563ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
122402025-02-28T12:14:10.563ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
122412025-02-28T12:14:10.563ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 106.356258ms iteration = 31 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
122422025-02-28T12:14:10.563ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
122432025-02-28T12:14:10.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
122442025-02-28T12:14:10.564ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122452025-02-28T12:14:10.564ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
122462025-02-28T12:14:10.564ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
122472025-02-28T12:14:10.564ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 107.932605ms iteration = 16 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
122482025-02-28T12:14:10.565ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122492025-02-28T12:14:10.567ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
122502025-02-28T12:14:10.567ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
122512025-02-28T12:14:10.567ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
122522025-02-28T12:14:10.567ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
122532025-02-28T12:14:10.567ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
122542025-02-28T12:14:10.567ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
122552025-02-28T12:14:10.568ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122562025-02-28T12:14:10.570ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
122572025-02-28T12:14:10.570ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
122582025-02-28T12:14:10.570ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
122592025-02-28T12:14:10.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
122602025-02-28T12:14:10.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
122612025-02-28T12:14:10.572ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
122622025-02-28T12:14:10.572ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
122632025-02-28T12:14:10.572ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122642025-02-28T12:14:10.574ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122652025-02-28T12:14:10.574ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
122662025-02-28T12:14:10.574ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
122672025-02-28T12:14:10.575ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
122682025-02-28T12:14:10.576ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
122692025-02-28T12:14:10.576ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122702025-02-28T12:14:10.576ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
122712025-02-28T12:14:10.577ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
122722025-02-28T12:14:10.577ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
122732025-02-28T12:14:10.577ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 120.52913ms iteration = 31 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
122742025-02-28T12:14:10.577ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
122752025-02-28T12:14:10.577ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
122762025-02-28T12:14:10.578ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
122772025-02-28T12:14:10.578ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
122782025-02-28T12:14:10.578ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
122792025-02-28T12:14:10.578ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
122802025-02-28T12:14:10.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
122812025-02-28T12:14:10.580ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122822025-02-28T12:14:10.582ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 125.462729ms iteration = 31 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
122832025-02-28T12:14:10.585ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
122842025-02-28T12:14:10.585ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
122852025-02-28T12:14:10.585ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
122862025-02-28T12:14:10.586ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
122872025-02-28T12:14:10.586ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122882025-02-28T12:14:10.589ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 96b8e143-6fd0-4d43-9fcf-f88c458c870d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
122892025-02-28T12:14:10.589ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
122902025-02-28T12:14:10.589ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-02-28 11:59:51.915187 UTC dns_group = internal generation = 1
122912025-02-28T12:14:10.589ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 136.64837ms iteration = 16 status = {"generation":1}
122922025-02-28T12:14:10.589ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
122932025-02-28T12:14:10.589ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
122942025-02-28T12:14:10.591ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
122952025-02-28T12:14:10.591ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
122962025-02-28T12:14:10.591ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
122972025-02-28T12:14:10.593ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
122982025-02-28T12:14:10.593ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
122992025-02-28T12:14:10.593ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 0 version = 2
123002025-02-28T12:14:10.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
123012025-02-28T12:14:10.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
123022025-02-28T12:14:10.598ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
123032025-02-28T12:14:10.598ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 141.610488ms iteration = 33 status = {"physical_disks_added":0}
123042025-02-28T12:14:10.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
123052025-02-28T12:14:10.599ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 141.291321ms iteration = 31 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
123062025-02-28T12:14:10.608ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123072025-02-28T12:14:10.613ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
123082025-02-28T12:14:10.613ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
123092025-02-28T12:14:10.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
123102025-02-28T12:14:10.616ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123112025-02-28T12:14:10.619ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
123122025-02-28T12:14:10.619ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
123132025-02-28T12:14:10.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
123142025-02-28T12:14:10.619ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
123152025-02-28T12:14:10.619ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 163.179409ms iteration = 16 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
123162025-02-28T12:14:10.625ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-28T11:59:08.719399Z, time_modified: 2025-02-28T11:59:08.719399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, db_silo: Silo { identity: SiloIdentity { id: fcd7a1e1-e386-45d7-bb38-cbddb94c7b10, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-28T11:59:52.459317Z, time_modified: 2025-02-28T11:59:52.459317Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682" }] }) }
123172025-02-28T12:14:10.625ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 173.121649ms iteration = 16 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]}},"default_endpoint":{"silo_id":"fcd7a1e1-e386-45d7-bb38-cbddb94c7b10","tls_certs":["eb6c45063d5444618948b97fc56ca0c1e4a35efb31b6e62d64d6cb4bb1324682"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
123182025-02-28T12:14:10.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123192025-02-28T12:14:10.632ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
123202025-02-28T12:14:10.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
123212025-02-28T12:14:10.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
123222025-02-28T12:14:10.655ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 196.602615ms iteration = 31 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
123232025-02-28T12:14:10.705ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
123242025-02-28T12:14:10.705ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
123252025-02-28T12:14:10.706ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8951b4c8-06a9-4a87-abfd-57036b894024", "content-length": "26", "date": "Fri, 28 Feb 2025 12:14:10 GMT"} })
123262025-02-28T12:14:10.706ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
123272025-02-28T12:14:10.708ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
123282025-02-28T12:14:10.805ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
123292025-02-28T12:14:10.805ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
123302025-02-28T12:14:10.805ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "33f9d9bf-bc17-4c4b-b585-1fb58e8c8b5f", "content-length": "26", "date": "Fri, 28 Feb 2025 12:14:10 GMT"} })
123312025-02-28T12:14:10.805ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
123322025-02-28T12:14:10.807ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
123332025-02-28T12:14:10.813ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
123342025-02-28T12:14:10.813ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
123352025-02-28T12:14:10.813ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b08b192-43b0-4b17-bdc0-43424a9dd14f", "content-length": "26", "date": "Fri, 28 Feb 2025 12:14:10 GMT"} })
123362025-02-28T12:14:10.813ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
123372025-02-28T12:14:10.815ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
123382025-02-28T12:14:10.834ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
123392025-02-28T12:14:10.834ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
123402025-02-28T12:14:10.835ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "df4f5287-cf0c-43c5-b332-a78ee9f871ef", "content-length": "372", "date": "Fri, 28 Feb 2025 12:14:10 GMT"} })
123412025-02-28T12:14:10.835ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
123422025-02-28T12:14:10.835ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
123432025-02-28T12:14:10.836ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123442025-02-28T12:14:10.836ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123452025-02-28T12:14:10.839ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
123462025-02-28T12:14:10.839ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
123472025-02-28T12:14:10.840ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
123482025-02-28T12:14:10.840ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
123492025-02-28T12:14:10.840ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
123502025-02-28T12:14:10.841ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
123512025-02-28T12:14:10.881ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
123522025-02-28T12:14:10.927ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
123532025-02-28T12:14:10.927ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "74e172cf-b2ab-484f-9fbf-47197db071e3", "date": "Fri, 28 Feb 2025 12:14:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
123542025-02-28T12:14:10.927ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
123552025-02-28T12:14:10.929ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "4a2692df-12d1-458e-9f32-8a5600763d99", "date": "Fri, 28 Feb 2025 12:14:10 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
123562025-02-28T12:14:10.929ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
123572025-02-28T12:14:10.930ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
123582025-02-28T12:14:10.930ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 478.327537ms iteration = 18 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
123592025-02-28T12:14:10.938ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
123602025-02-28T12:14:10.939ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "16077613-2171-4057-853a-55fff3480501", "content-length": "691", "date": "Fri, 28 Feb 2025 12:14:10 GMT"} })
123612025-02-28T12:14:10.939ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
123622025-02-28T12:14:10.939ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
123632025-02-28T12:14:10.939ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
123642025-02-28T12:14:10.941ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 484.521007ms iteration = 31 status = {}
123652025-02-28T12:14:10.949ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
123662025-02-28T12:14:10.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
123672025-02-28T12:14:10.951ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
123682025-02-28T12:14:10.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
123692025-02-28T12:14:10.953ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 496.352942ms iteration = 31 status = 0
123702025-02-28T12:14:10.953ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
123712025-02-28T12:14:10.953ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 496.939626ms iteration = 6 status = {"blueprint_id":"7a676565-49af-418f-9d1f-e6c070e71120","inventory_collection_id":"a84bcc24-1946-49c1-8d43-7345b4933ff0","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
123722025-02-28T12:14:10.955ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123732025-02-28T12:14:10.955ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 1b3859ef-c76f-47d2-a761-cae84cfbe0de dns_zone_name = oxide.test found_so_far = 1 version = 2
123742025-02-28T12:14:10.955ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
123752025-02-28T12:14:10.955ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-02-28 11:59:52.542241 UTC dns_group = external generation = 2
123762025-02-28T12:14:10.955ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 503.550556ms iteration = 16 status = {"generation":2}
123772025-02-28T12:14:10.956ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
123782025-02-28T12:14:10.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
123792025-02-28T12:14:10.958ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
123802025-02-28T12:14:10.958ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
123812025-02-28T12:14:10.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
123822025-02-28T12:14:10.974ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 521.995491ms iteration = 31 status = {}
123832025-02-28T12:14:10.983ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
123842025-02-28T12:14:10.984ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "35a0ec0a-7f60-46fa-af22-1fd7358287e9", "content-length": "2", "date": "Fri, 28 Feb 2025 12:14:10 GMT"} })
123852025-02-28T12:14:10.984ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:619 sled = http://[fd00:1122:3344:101::1]:12345
123862025-02-28T12:14:10.986ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 527.679408ms iteration = 4 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":3,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"operation":"list","target_sled":"50ac5f18-4748-43d4-a4ac-8d2114194efc","time":"2025-02-28T12:04:11.359459081Z"},{"operation":"list","target_sled":"50ac5f18-4748-43d4-a4ac-8d2114194efc","time":"2025-02-28T12:09:11.028643484Z"},{"operation":"list","target_sled":"50ac5f18-4748-43d4-a4ac-8d2114194efc","time":"2025-02-28T12:14:10.984680514Z"}]}
123872025-02-28T12:14:10.990ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
123882025-02-28T12:14:10.990ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "e09f3145-e211-4846-bc2a-517a4e754614", "content-length": "2", "date": "Fri, 28 Feb 2025 12:14:10 GMT"} })
123892025-02-28T12:14:10.992ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
123902025-02-28T12:14:10.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
123912025-02-28T12:14:10.995ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 543.159362ms iteration = 31 status = null
123922025-02-28T12:14:11.003ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
123932025-02-28T12:14:11.003ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 546.348689ms iteration = 31 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
123942025-02-28T12:14:11.014ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
123952025-02-28T12:14:11.015ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "bf68b6d1-05ae-49ff-8a09-688355d2ff8c", "date": "Fri, 28 Feb 2025 12:14:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
123962025-02-28T12:14:11.015ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
123972025-02-28T12:14:11.016ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "75e4caee-e2e4-4f83-88d6-10316546ba7d", "date": "Fri, 28 Feb 2025 12:14:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
123982025-02-28T12:14:11.016ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
123992025-02-28T12:14:11.019ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "e944e30b-2bce-4cc5-9b8b-87c971a8a0c0", "date": "Fri, 28 Feb 2025 12:14:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
124002025-02-28T12:14:11.019ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
124012025-02-28T12:14:11.020ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
124022025-02-28T12:14:11.020ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 568.383351ms iteration = 18 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
124032025-02-28T12:14:11.023ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124042025-02-28T12:14:11.027ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
124052025-02-28T12:14:11.027ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124062025-02-28T12:14:11.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124072025-02-28T12:14:11.031ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124082025-02-28T12:14:11.031ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
124092025-02-28T12:14:11.031ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
124102025-02-28T12:14:11.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124112025-02-28T12:14:11.034ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "df9de48a-cd97-4b53-b317-5c152138c7b2", "date": "Fri, 28 Feb 2025 12:14:10 GMT"} })
124122025-02-28T12:14:11.034ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
124132025-02-28T12:14:11.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
124142025-02-28T12:14:11.035ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124152025-02-28T12:14:11.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124162025-02-28T12:14:11.037ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
124172025-02-28T12:14:11.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
124182025-02-28T12:14:11.038ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124192025-02-28T12:14:11.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124202025-02-28T12:14:11.039ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124212025-02-28T12:14:11.039ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
124222025-02-28T12:14:11.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
124232025-02-28T12:14:11.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124242025-02-28T12:14:11.041ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124252025-02-28T12:14:11.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124262025-02-28T12:14:11.043ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124272025-02-28T12:14:11.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124282025-02-28T12:14:11.043ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
124292025-02-28T12:14:11.043ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
124302025-02-28T12:14:11.043ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "73be9d28-c553-401d-b34e-2c791eca7745", "date": "Fri, 28 Feb 2025 12:14:10 GMT"} })
124312025-02-28T12:14:11.045ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124322025-02-28T12:14:11.046ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 589.371448ms iteration = 31 status = {}
124332025-02-28T12:14:11.048ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
124342025-02-28T12:14:11.049ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124352025-02-28T12:14:11.049ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124362025-02-28T12:14:11.051ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124372025-02-28T12:14:11.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124382025-02-28T12:14:11.052ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124392025-02-28T12:14:11.053ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124402025-02-28T12:14:11.053ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
124412025-02-28T12:14:11.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
124422025-02-28T12:14:11.054ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124432025-02-28T12:14:11.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124442025-02-28T12:14:11.056ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
124452025-02-28T12:14:11.056ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
124462025-02-28T12:14:11.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
124472025-02-28T12:14:11.057ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124482025-02-28T12:14:11.057ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124492025-02-28T12:14:11.066ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
124502025-02-28T12:14:11.066ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "517c5566-aabf-48fd-825a-c167cd89f5d5", "content-length": "134", "date": "Fri, 28 Feb 2025 12:14:10 GMT"} })
124512025-02-28T12:14:11.066ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
124522025-02-28T12:14:11.066ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
124532025-02-28T12:14:11.066ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
124542025-02-28T12:14:11.066ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
124552025-02-28T12:14:11.066ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
124562025-02-28T12:14:11.066ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
124572025-02-28T12:14:11.066ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
124582025-02-28T12:14:11.066ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
124592025-02-28T12:14:11.067ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
124602025-02-28T12:14:11.067ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
124612025-02-28T12:14:11.067ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
124622025-02-28T12:14:11.067ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
124632025-02-28T12:14:11.067ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
124642025-02-28T12:14:11.069ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
124652025-02-28T12:14:11.069ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "98fe8f44-0dd9-49d1-a488-5a8b63d00014", "content-length": "2", "date": "Fri, 28 Feb 2025 12:14:10 GMT"} })
124662025-02-28T12:14:11.069ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
124672025-02-28T12:14:11.069ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd12497d-d8f0-4d6a-a5a7-284e409d9676", "content-length": "51", "date": "Fri, 28 Feb 2025 12:14:10 GMT"} })
124682025-02-28T12:14:11.069ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
124692025-02-28T12:14:11.069ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
124702025-02-28T12:14:11.093ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124712025-02-28T12:14:11.093ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124722025-02-28T12:14:11.095ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124732025-02-28T12:14:11.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124742025-02-28T12:14:11.097ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124752025-02-28T12:14:11.101ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
124762025-02-28T12:14:11.101ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124772025-02-28T12:14:11.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124782025-02-28T12:14:11.107ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124792025-02-28T12:14:11.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124802025-02-28T12:14:11.108ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124812025-02-28T12:14:11.109ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124822025-02-28T12:14:11.110ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124832025-02-28T12:14:11.114ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
124842025-02-28T12:14:11.114ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124852025-02-28T12:14:11.114ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124862025-02-28T12:14:11.120ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124872025-02-28T12:14:11.120ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124882025-02-28T12:14:11.122ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124892025-02-28T12:14:11.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124902025-02-28T12:14:11.125ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124912025-02-28T12:14:11.125ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124922025-02-28T12:14:11.127ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124932025-02-28T12:14:11.128ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124942025-02-28T12:14:11.138ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124952025-02-28T12:14:11.139ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124962025-02-28T12:14:11.140ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124972025-02-28T12:14:11.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
124982025-02-28T12:14:11.142ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
124992025-02-28T12:14:11.143ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
125002025-02-28T12:14:11.144ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125012025-02-28T12:14:11.148ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
125022025-02-28T12:14:11.148ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
125032025-02-28T12:14:11.148ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
125042025-02-28T12:14:11.151ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
125052025-02-28T12:14:11.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
125062025-02-28T12:14:11.154ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
125072025-02-28T12:14:11.154ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
125082025-02-28T12:14:11.155ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125092025-02-28T12:14:11.159ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
125102025-02-28T12:14:11.159ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
125112025-02-28T12:14:11.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
125122025-02-28T12:14:11.166ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: efece645-4fc7-4f5a-900b-9b0d58342421, kind: Service { id: db281835-e35c-480d-b1a2-07f321816d55 }, name: Name("external-dns-db281835-e35c-480d-b1a2-07f321816d55"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 88])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fb9b46af-fe90-4b4d-b831-9e9a193e0996, kind: Service { id: 5cefabf1-6831-46ff-98eb-1a3b5abad8b9 }, name: Name("external-dns-5cefabf1-6831-46ff-98eb-1a3b5abad8b9"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 243, 164])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 08731947-b4fa-4741-bf20-8ff14850f21a, kind: Service { id: 3bbfd7d6-3f72-4034-ad35-e956e1c12923 }, name: Name("nexus-3bbfd7d6-3f72-4034-ad35-e956e1c12923"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 4814ec6a-93e1-40fa-ba73-651e41bc6ee3, kind: Service { id: c1b34bc5-1c2f-4d3c-808c-b7a441aac295 }, name: Name("nexus-c1b34bc5-1c2f-4d3c-808c-b7a441aac295"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 152])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e56f48b3-247c-430d-8e79-57098c59bab5, kind: Service { id: e594478d-1d06-4692-a5d7-cded9c00f6d4 }, name: Name("nexus-e594478d-1d06-4692-a5d7-cded9c00f6d4"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 182, 65])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
125132025-02-28T12:14:11.166ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
125142025-02-28T12:14:11.167ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
125152025-02-28T12:14:11.168ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125162025-02-28T12:14:11.170ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
125172025-02-28T12:14:11.170ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
125182025-02-28T12:14:11.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
125192025-02-28T12:14:11.172ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
125202025-02-28T12:14:11.172ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: efece645-4fc7-4f5a-900b-9b0d58342421, kind: Service { id: db281835-e35c-480d-b1a2-07f321816d55 }, name: Name("external-dns-db281835-e35c-480d-b1a2-07f321816d55"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 88])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: fb9b46af-fe90-4b4d-b831-9e9a193e0996, kind: Service { id: 5cefabf1-6831-46ff-98eb-1a3b5abad8b9 }, name: Name("external-dns-5cefabf1-6831-46ff-98eb-1a3b5abad8b9"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 243, 164])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 08731947-b4fa-4741-bf20-8ff14850f21a, kind: Service { id: 3bbfd7d6-3f72-4034-ad35-e956e1c12923 }, name: Name("nexus-3bbfd7d6-3f72-4034-ad35-e956e1c12923"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 4814ec6a-93e1-40fa-ba73-651e41bc6ee3, kind: Service { id: c1b34bc5-1c2f-4d3c-808c-b7a441aac295 }, name: Name("nexus-c1b34bc5-1c2f-4d3c-808c-b7a441aac295"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 152])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e56f48b3-247c-430d-8e79-57098c59bab5, kind: Service { id: e594478d-1d06-4692-a5d7-cded9c00f6d4 }, name: Name("nexus-e594478d-1d06-4692-a5d7-cded9c00f6d4"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 182, 65])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
125212025-02-28T12:14:11.172ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
125222025-02-28T12:14:11.195ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "7953b396-6c06-44b3-ac18-278a215cadec", "date": "Fri, 28 Feb 2025 12:14:10 GMT"} })
125232025-02-28T12:14:11.195ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
125242025-02-28T12:14:11.195ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
125252025-02-28T12:14:11.196ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1bf4be2-40c7-4a88-a2ab-28e928b650f2", "content-length": "682", "date": "Fri, 28 Feb 2025 12:14:11 GMT"} })
125262025-02-28T12:14:11.196ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
125272025-02-28T12:14:11.196ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
125282025-02-28T12:14:11.200ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
125292025-02-28T12:14:11.200ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
125302025-02-28T12:14:11.204ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
125312025-02-28T12:14:11.204ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
125322025-02-28T12:14:11.206ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
125332025-02-28T12:14:11.206ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
125342025-02-28T12:14:11.214ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 757.548648ms iteration = 31 status = {}
125352025-02-28T12:14:11.303ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 50ac5f18-4748-43d4-a4ac-8d2114194efc, time_created: 2025-02-28T11:59:46.871334Z, time_modified: 2025-02-28T11:59:46.871334Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::1:0, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
125362025-02-28T12:14:11.303ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
125372025-02-28T12:14:11.303ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
125382025-02-28T12:14:11.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
125392025-02-28T12:14:11.306ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
125402025-02-28T12:14:11.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
125412025-02-28T12:14:11.308ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125422025-02-28T12:14:11.310ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
125432025-02-28T12:14:11.341ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
125442025-02-28T12:14:11.345ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "71035cb2-b973-4c8d-b968-2d803dde6d68", "date": "Fri, 28 Feb 2025 12:14:11 GMT"} })
125452025-02-28T12:14:11.348ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
125462025-02-28T12:14:11.348ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 891.280936ms
125472025-02-28T12:14:11.348ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 891.290397ms iteration = 4 status = {}
125482025-02-28T12:14:14.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 306 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = c5579859-7cca-48f2-b1e6-3f553b94c336 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
125492025-02-28T12:14:20.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 92 reason = Timeout
125502025-02-28T12:14:20.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
125512025-02-28T12:14:20.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
125522025-02-28T12:14:20.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 92 reason = Timeout
125532025-02-28T12:14:20.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
125542025-02-28T12:14:20.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125552025-02-28T12:14:20.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
125562025-02-28T12:14:20.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
125572025-02-28T12:14:20.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
125582025-02-28T12:14:20.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
125592025-02-28T12:14:20.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
125602025-02-28T12:14:20.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
125612025-02-28T12:14:20.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125622025-02-28T12:14:20.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
125632025-02-28T12:14:20.467ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
125642025-02-28T12:14:20.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
125652025-02-28T12:14:20.479ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
125662025-02-28T12:14:20.479ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.716062ms iteration = 92 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
125672025-02-28T12:14:20.487ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
125682025-02-28T12:14:20.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
125692025-02-28T12:14:20.518ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 61.249618ms iteration = 92 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
125702025-02-28T12:14:24.252ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42136 req_id = 53f15b16-77b0-4fcf-a4e8-8f7105a26c1e roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
125712025-02-28T12:14:24.253ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42136 req_id = 53f15b16-77b0-4fcf-a4e8-8f7105a26c1e resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
125722025-02-28T12:14:24.255ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3174 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42136 req_id = 53f15b16-77b0-4fcf-a4e8-8f7105a26c1e response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?sort_by=id_ascending
125732025-02-28T12:14:24.255ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42136 req_id = 183fc887-9741-4edf-9d2f-92bf6fd060d2 roles = RoleSet { roles: {} } uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
125742025-02-28T12:14:24.256ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42136 req_id = 183fc887-9741-4edf-9d2f-92bf6fd060d2 resource = Database result = Ok(()) uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
125752025-02-28T12:14:24.258ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2702 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:42136 req_id = 183fc887-9741-4edf-9d2f-92bf6fd060d2 response_code = 200 uri = /metrics/collectors/a06e31c0-a3bc-4a7a-a9aa-51a6448e3e79/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlOGMxNzFmNS0yY2EyLTQ0ZjktYjZkOS1iZDJiNGRjNTdlMmIifX0%3D
125762025-02-28T12:14:24.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 309 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = c0e3e447-d748-46a6-9ec6-5a717a8426b7 response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
125772025-02-28T12:14:30.455ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 93 reason = Timeout
125782025-02-28T12:14:30.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
125792025-02-28T12:14:30.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
125802025-02-28T12:14:30.457ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 93 reason = Timeout
125812025-02-28T12:14:30.457ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
125822025-02-28T12:14:30.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
125832025-02-28T12:14:30.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125842025-02-28T12:14:30.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
125852025-02-28T12:14:30.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
125862025-02-28T12:14:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
125872025-02-28T12:14:30.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
125882025-02-28T12:14:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
125892025-02-28T12:14:30.461ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.174981ms iteration = 93 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
125902025-02-28T12:14:30.462ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
125912025-02-28T12:14:30.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
125922025-02-28T12:14:30.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125932025-02-28T12:14:30.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
125942025-02-28T12:14:30.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
125952025-02-28T12:14:30.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
125962025-02-28T12:14:30.476ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
125972025-02-28T12:14:30.476ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.201772ms iteration = 93 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
125982025-02-28T12:14:34.259ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 420 local_addr = [fd00:1122:3344:101::a]:56450 method = GET remote_addr = [fd00:1122:3344:101::d]:49165 req_id = f54f14c3-2c2e-4411-9aba-435b3203573c response_code = 200 uri = /c1b34bc5-1c2f-4d3c-808c-b7a441aac295
125992025-02-28T12:14:40.451ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 32 reason = Timeout
126002025-02-28T12:14:40.452ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
126012025-02-28T12:14:40.452ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 32 reason = Timeout
126022025-02-28T12:14:40.452ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
126032025-02-28T12:14:40.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
126042025-02-28T12:14:40.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
126052025-02-28T12:14:40.454ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1715, tv_nsec: 92507255 } })
126062025-02-28T12:14:40.454ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 94 reason = Timeout
126072025-02-28T12:14:40.454ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 32 reason = Timeout
126082025-02-28T12:14:40.454ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
126092025-02-28T12:14:40.454ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
126102025-02-28T12:14:40.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
126112025-02-28T12:14:40.455ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1715, tv_nsec: 93886379 } })
126122025-02-28T12:14:40.455ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
126132025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126142025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 94 reason = Timeout
126152025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
126162025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 32 reason = Timeout
126172025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
126182025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 32 reason = Timeout
126192025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
126202025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
126212025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 32 reason = Timeout
126222025-02-28T12:14:40.456ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
126232025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 32 reason = Timeout
126242025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
126252025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
126262025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 32 reason = Timeout
126272025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 32 reason = Timeout
126282025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
126292025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
126302025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 32 reason = Timeout
126312025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
126322025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 34 reason = Timeout
126332025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
126342025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 32 reason = Timeout
126352025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
126362025-02-28T12:14:40.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
126372025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
126382025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
126392025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
126402025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
126412025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
126422025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
126432025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
126442025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
126452025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
126462025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 32 reason = Timeout
126472025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 32 reason = Timeout
126482025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
126492025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
126502025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 32 reason = Timeout
126512025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
126522025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 32 reason = Timeout
126532025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
126542025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 32 reason = Timeout
126552025-02-28T12:14:40.457ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
126562025-02-28T12:14:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
126572025-02-28T12:14:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
126582025-02-28T12:14:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
126592025-02-28T12:14:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
126602025-02-28T12:14:40.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
126612025-02-28T12:14:40.458ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.574636ms iteration = 32 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
126622025-02-28T12:14:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
126632025-02-28T12:14:40.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
126642025-02-28T12:14:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
126652025-02-28T12:14:40.459ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
126662025-02-28T12:14:40.459ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
126672025-02-28T12:14:40.459ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
126682025-02-28T12:14:40.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
126692025-02-28T12:14:40.460ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126702025-02-28T12:14:40.461ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126712025-02-28T12:14:40.462ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126722025-02-28T12:14:40.462ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126732025-02-28T12:14:40.462ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126742025-02-28T12:14:40.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
126752025-02-28T12:14:40.462ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
126762025-02-28T12:14:40.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
126772025-02-28T12:14:40.464ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126782025-02-28T12:14:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
126792025-02-28T12:14:40.464ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
126802025-02-28T12:14:40.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
126812025-02-28T12:14:40.464ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
126822025-02-28T12:14:40.465ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
126832025-02-28T12:14:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
126842025-02-28T12:14:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
126852025-02-28T12:14:40.465ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
126862025-02-28T12:14:40.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
126872025-02-28T12:14:40.465ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
126882025-02-28T12:14:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
126892025-02-28T12:14:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
126902025-02-28T12:14:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
126912025-02-28T12:14:40.466ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
126922025-02-28T12:14:40.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
126932025-02-28T12:14:40.470ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
126942025-02-28T12:14:40.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
126952025-02-28T12:14:40.471ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
126962025-02-28T12:14:40.471ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-50ac5f18-4748-43d4-a4ac-8d2114194efc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1715, tv_nsec: 110061529 } })
126972025-02-28T12:14:40.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
126982025-02-28T12:14:40.472ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
126992025-02-28T12:14:40.478ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
127002025-02-28T12:14:40.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
127012025-02-28T12:14:40.504ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
127022025-02-28T12:14:40.504ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
127032025-02-28T12:14:40.508ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 50.732065ms iteration = 32 status = {"errors":[],"finish_invoked_ok":[]}
127042025-02-28T12:14:40.510ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
127052025-02-28T12:14:40.510ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
127062025-02-28T12:14:40.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
127072025-02-28T12:14:40.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
127082025-02-28T12:14:40.513ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127092025-02-28T12:14:40.515ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
127102025-02-28T12:14:40.515ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
127112025-02-28T12:14:40.515ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
127122025-02-28T12:14:40.515ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
127132025-02-28T12:14:40.515ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
127142025-02-28T12:14:40.515ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
127152025-02-28T12:14:40.515ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 58.946763ms iteration = 32 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
127162025-02-28T12:14:40.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
127172025-02-28T12:14:40.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
127182025-02-28T12:14:40.516ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127192025-02-28T12:14:40.518ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
127202025-02-28T12:14:40.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
127212025-02-28T12:14:40.519ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
127222025-02-28T12:14:40.519ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
127232025-02-28T12:14:40.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
127242025-02-28T12:14:40.519ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127252025-02-28T12:14:40.520ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
127262025-02-28T12:14:40.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
127272025-02-28T12:14:40.522ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 64.173969ms iteration = 32 status = {"errors":[],"garbage_collect_requested":[]}
127282025-02-28T12:14:40.523ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 7a676565-49af-418f-9d1f-e6c070e71120, lookup_type: ById(7a676565-49af-418f-9d1f-e6c070e71120) } result = Ok(())
127292025-02-28T12:14:40.523ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
127302025-02-28T12:14:40.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
127312025-02-28T12:14:40.523ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
127322025-02-28T12:14:40.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
127332025-02-28T12:14:40.526ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
127342025-02-28T12:14:40.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
127352025-02-28T12:14:40.527ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
127362025-02-28T12:14:40.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
127372025-02-28T12:14:40.529ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 71.338332ms iteration = 32 status = {"errors":[],"requests_created_ok":[]}
127382025-02-28T12:14:40.545ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
127392025-02-28T12:14:40.545ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
127402025-02-28T12:14:40.545ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "27d07a45-8edb-40f6-84a8-9b4289adc67b", "content-length": "26", "date": "Fri, 28 Feb 2025 12:14:40 GMT"} })
127412025-02-28T12:14:40.545ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
127422025-02-28T12:14:40.547ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
127432025-02-28T12:14:40.550ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 93.373614ms iteration = 32 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
127442025-02-28T12:14:40.555ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
127452025-02-28T12:14:40.555ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
127462025-02-28T12:14:40.555ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a19551d9-486a-427c-bbb7-97d2f08919f6", "content-length": "26", "date": "Fri, 28 Feb 2025 12:14:40 GMT"} })
127472025-02-28T12:14:40.555ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
127482025-02-28T12:14:40.556ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 100.186139ms iteration = 94 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
127492025-02-28T12:14:40.556ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
127502025-02-28T12:14:40.557ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
127512025-02-28T12:14:40.557ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
127522025-02-28T12:14:40.557ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 100.454555ms iteration = 32 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
127532025-02-28T12:14:40.578ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef zone_address = fd00:1122:3344:101::2
127542025-02-28T12:14:40.578ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
127552025-02-28T12:14:40.579ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b16adeb-5032-49be-8bc1-99de1d8b5ed7", "content-length": "26", "date": "Fri, 28 Feb 2025 12:14:40 GMT"} })
127562025-02-28T12:14:40.579ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
127572025-02-28T12:14:40.580ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
127582025-02-28T12:14:40.683ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
127592025-02-28T12:14:40.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
127602025-02-28T12:14:40.691ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 239.768318ms iteration = 32 status = {}
127612025-02-28T12:14:40.725ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
127622025-02-28T12:14:40.726ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "70651fd2-1115-476b-8585-e76ed52d7336", "content-length": "2", "date": "Fri, 28 Feb 2025 12:14:40 GMT"} })
127632025-02-28T12:14:40.729ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
127642025-02-28T12:14:40.729ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
127652025-02-28T12:14:40.732ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
127662025-02-28T12:14:40.732ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
127672025-02-28T12:14:40.732ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "73a0d851-f9a0-4f27-ab16-f50e6de29980", "content-length": "372", "date": "Fri, 28 Feb 2025 12:14:40 GMT"} })
127682025-02-28T12:14:40.732ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
127692025-02-28T12:14:40.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
127702025-02-28T12:14:40.738ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127712025-02-28T12:14:40.740ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 288.581256ms iteration = 32 status = null
127722025-02-28T12:14:40.741ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
127732025-02-28T12:14:40.741ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
127742025-02-28T12:14:40.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
127752025-02-28T12:14:40.743ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127762025-02-28T12:14:40.747ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
127772025-02-28T12:14:40.747ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
127782025-02-28T12:14:40.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
127792025-02-28T12:14:40.753ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 7a676565-49af-418f-9d1f-e6c070e71120 original_time_created = 2025-02-28 11:59:04.656609 UTC target_id = 7a676565-49af-418f-9d1f-e6c070e71120 time_created = 2025-02-28 11:59:04.656609 UTC
127802025-02-28T12:14:40.753ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 298.756752ms iteration = 94 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"7a676565-49af-418f-9d1f-e6c070e71120","time_created":"2025-02-28T11:59:04.656609Z"}
127812025-02-28T12:14:40.797ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 339.334661ms iteration = 32 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
127822025-02-28T12:14:40.801ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
127832025-02-28T12:14:40.803ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
127842025-02-28T12:14:40.804ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127852025-02-28T12:14:40.806ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
127862025-02-28T12:14:40.806ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
127872025-02-28T12:14:40.806ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
127882025-02-28T12:14:40.807ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
127892025-02-28T12:14:40.808ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
127902025-02-28T12:14:40.810ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127912025-02-28T12:14:40.810ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 353.820192ms iteration = 34 status = {"physical_disks_added":0}
127922025-02-28T12:14:40.811ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 6bd65cbd-8319-4a4f-9bd5-db169ebdf54f, rack_id: 53c1bff6-f633-4d8b-8c3f-66c54025dbef, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(03e637f9-2761-45fb-81ee-52d8c7fe9351) } rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
127932025-02-28T12:14:40.811ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
127942025-02-28T12:14:40.812ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
127952025-02-28T12:14:40.814ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 50ac5f18-4748-43d4-a4ac-8d2114194efc, lookup_type: ById(50ac5f18-4748-43d4-a4ac-8d2114194efc) } result = Ok(())
127962025-02-28T12:14:40.814ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
127972025-02-28T12:14:40.814ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
127982025-02-28T12:14:40.817ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 360.696868ms iteration = 32 status = 0
127992025-02-28T12:14:40.833ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
128002025-02-28T12:14:40.833ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
128012025-02-28T12:14:40.833ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "d3df03b2-133f-43fd-be5d-fc42fc09e6ca", "date": "Fri, 28 Feb 2025 12:14:40 GMT"} })
128022025-02-28T12:14:40.833ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
128032025-02-28T12:14:40.834ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
128042025-02-28T12:14:40.834ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
128052025-02-28T12:14:40.834ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d1cfefb-00d4-4b4b-a24f-85d223c1358f", "content-length": "134", "date": "Fri, 28 Feb 2025 12:14:40 GMT"} })
128062025-02-28T12:14:40.834ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
128072025-02-28T12:14:40.834ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 03e637f9-2761-45fb-81ee-52d8c7fe9351
128082025-02-28T12:14:40.835ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
128092025-02-28T12:14:40.835ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
128102025-02-28T12:14:40.835ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
128112025-02-28T12:14:40.835ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
128122025-02-28T12:14:40.835ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
128132025-02-28T12:14:40.835ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
128142025-02-28T12:14:40.835ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
128152025-02-28T12:14:40.835ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
128162025-02-28T12:14:40.835ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef
128172025-02-28T12:14:40.835ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
128182025-02-28T12:14:40.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
128192025-02-28T12:14:40.837ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
128202025-02-28T12:14:40.837ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
128212025-02-28T12:14:40.840ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
128222025-02-28T12:14:40.840ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9fc8428-f862-4962-aab6-c0e4468e110b", "content-length": "2", "date": "Fri, 28 Feb 2025 12:14:40 GMT"} })
128232025-02-28T12:14:40.840ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
128242025-02-28T12:14:40.840ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "a57c5b61-753e-4b20-b5e1-96a98741acc9", "content-length": "51", "date": "Fri, 28 Feb 2025 12:14:40 GMT"} })
128252025-02-28T12:14:40.840ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch_location = Switch0
128262025-02-28T12:14:40.840ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
128272025-02-28T12:14:40.840ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
128282025-02-28T12:14:40.841ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
128292025-02-28T12:14:40.845ZINFOnexus (BackgroundTasks): VPC route manager sled 50ac5f18-4748-43d4-a4ac-8d2114194efc push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
128302025-02-28T12:14:40.845ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
128312025-02-28T12:14:40.848ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 390.23611ms iteration = 32 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
128322025-02-28T12:14:40.869ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
128332025-02-28T12:14:40.869ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 413.036042ms iteration = 32 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
128342025-02-28T12:14:40.902ZDEBGnexus (BackgroundTasks): client request SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
128352025-02-28T12:14:40.988ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "d03d8d00-dc4a-43df-be0f-fbf0f52b540d", "date": "Fri, 28 Feb 2025 12:14:40 GMT"} })
128362025-02-28T12:14:40.988ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef switch = Switch0
128372025-02-28T12:14:40.988ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
128382025-02-28T12:14:40.988ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 53c1bff6-f633-4d8b-8c3f-66c54025dbef result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c83184e4-7895-476b-99b3-f85df884868e", "content-length": "682", "date": "Fri, 28 Feb 2025 12:14:40 GMT"} })
128392025-02-28T12:14:40.988ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
128402025-02-28T12:14:40.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
128412025-02-28T12:14:40.989ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "80325517-364a-403b-a577-182636af19cb", "date": "Fri, 28 Feb 2025 12:14:40 GMT"} })
128422025-02-28T12:14:40.989ZDEBGnexus (BackgroundTasks): client response SledAgent = 50ac5f18-4748-43d4-a4ac-8d2114194efc background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f13a1eb-df61-48b8-9eb9-187359ad445d", "content-length": "691", "date": "Fri, 28 Feb 2025 12:14:40 GMT"} })
128432025-02-28T12:14:40.989ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
128442025-02-28T12:14:40.989ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
128452025-02-28T12:14:40.989ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
128462025-02-28T12:14:40.991ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 534.580256ms iteration = 32 status = {}
128472025-02-28T12:14:40.991ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 534.80556ms iteration = 32 status = {}
128482025-02-28T12:14:40.994ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
128492025-02-28T12:14:40.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
128502025-02-28T12:14:40.998ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
128512025-02-28T12:14:40.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
128522025-02-28T12:14:41.001ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
128532025-02-28T12:14:41.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
128542025-02-28T12:14:41.016ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 559.791824ms iteration = 32 status = {}