Max level shown:
12025-03-02T05:00:10.438ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 26 reason = Timeout
22025-03-02T05:00:10.438ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32025-03-02T05:00:10.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42025-03-02T05:00:10.441ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 26 reason = Timeout
52025-03-02T05:00:10.441ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62025-03-02T05:00:10.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
72025-03-02T05:00:10.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82025-03-02T05:00:10.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92025-03-02T05:00:10.468ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102025-03-02T05:00:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
112025-03-02T05:00:10.471ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
122025-03-02T05:00:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
132025-03-02T05:00:10.473ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.5473ms iteration = 26 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
142025-03-02T05:00:10.473ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
152025-03-02T05:00:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
162025-03-02T05:00:10.474ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
172025-03-02T05:00:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
182025-03-02T05:00:10.477ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
192025-03-02T05:00:10.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
202025-03-02T05:00:10.487ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
212025-03-02T05:00:10.487ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 49.429044ms iteration = 26 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
222025-03-02T05:00:12.901ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 262 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 8379a0a0-d74c-4cf7-9530-9c98d5570acd response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
232025-03-02T05:00:20.438ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 27 reason = Timeout
242025-03-02T05:00:20.438ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
252025-03-02T05:00:20.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
262025-03-02T05:00:20.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
272025-03-02T05:00:20.441ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 27 reason = Timeout
282025-03-02T05:00:20.441ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
292025-03-02T05:00:20.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
302025-03-02T05:00:20.442ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
312025-03-02T05:00:20.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
322025-03-02T05:00:20.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
332025-03-02T05:00:20.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
342025-03-02T05:00:20.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
352025-03-02T05:00:20.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
362025-03-02T05:00:20.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
372025-03-02T05:00:20.445ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.070473ms iteration = 27 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
382025-03-02T05:00:20.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
392025-03-02T05:00:20.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
402025-03-02T05:00:20.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
412025-03-02T05:00:20.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
422025-03-02T05:00:20.460ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
432025-03-02T05:00:20.460ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.199329ms iteration = 27 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
442025-03-02T05:00:22.900ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 259 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 8271dee6-a67f-4615-a7eb-3cd32ca40698 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
452025-03-02T05:00:27.894ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = c125b171-0605-4f50-9d44-2cdc8085286e roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
462025-03-02T05:00:27.894ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = c125b171-0605-4f50-9d44-2cdc8085286e resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
472025-03-02T05:00:27.896ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2044 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = c125b171-0605-4f50-9d44-2cdc8085286e response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
482025-03-02T05:00:27.896ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = 61274c80-d78b-4e82-acc0-cc6bf416e3d2 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
492025-03-02T05:00:27.896ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = 61274c80-d78b-4e82-acc0-cc6bf416e3d2 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
502025-03-02T05:00:27.898ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2261 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = 61274c80-d78b-4e82-acc0-cc6bf416e3d2 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
512025-03-02T05:00:30.435ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 10 reason = Timeout
522025-03-02T05:00:30.435ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 10 reason = Timeout
532025-03-02T05:00:30.435ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
542025-03-02T05:00:30.435ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 10 reason = Timeout
552025-03-02T05:00:30.435ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
562025-03-02T05:00:30.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
572025-03-02T05:00:30.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
582025-03-02T05:00:30.438ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3359, tv_nsec: 668690624 } })
592025-03-02T05:00:30.438ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 10 reason = Timeout
602025-03-02T05:00:30.438ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 28 reason = Timeout
612025-03-02T05:00:30.438ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 10 reason = Timeout
622025-03-02T05:00:30.438ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 11 reason = Timeout
632025-03-02T05:00:30.438ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
642025-03-02T05:00:30.438ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
652025-03-02T05:00:30.438ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
662025-03-02T05:00:30.438ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
672025-03-02T05:00:30.438ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3359, tv_nsec: 668923787 } })
682025-03-02T05:00:30.438ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
692025-03-02T05:00:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
702025-03-02T05:00:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
712025-03-02T05:00:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
722025-03-02T05:00:30.438ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
732025-03-02T05:00:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
742025-03-02T05:00:30.439ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.606607ms iteration = 10 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
752025-03-02T05:00:30.440ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
762025-03-02T05:00:30.440ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
772025-03-02T05:00:30.441ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
782025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 10 reason = Timeout
792025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
802025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 28 reason = Timeout
812025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 10 reason = Timeout
822025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
832025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
842025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 10 reason = Timeout
852025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
862025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 10 reason = Timeout
872025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
882025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 10 reason = Timeout
892025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
902025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 10 reason = Timeout
912025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 10 reason = Timeout
922025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
932025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
942025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 10 reason = Timeout
952025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
962025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 10 reason = Timeout
972025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
982025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
992025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 10 reason = Timeout
1002025-03-02T05:00:30.442ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
1012025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1022025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 10 reason = Timeout
1032025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
1042025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1052025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
1062025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1072025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1082025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
1092025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1102025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
1112025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1122025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1132025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1142025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
1152025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1162025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
1172025-03-02T05:00:30.442ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1182025-03-02T05:00:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1192025-03-02T05:00:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
1202025-03-02T05:00:30.444ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1212025-03-02T05:00:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1222025-03-02T05:00:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
1232025-03-02T05:00:30.444ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1242025-03-02T05:00:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1252025-03-02T05:00:30.447ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1262025-03-02T05:00:30.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
1272025-03-02T05:00:30.449ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
1282025-03-02T05:00:30.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1292025-03-02T05:00:30.460ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1302025-03-02T05:00:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1312025-03-02T05:00:30.460ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1322025-03-02T05:00:30.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1332025-03-02T05:00:30.473ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1342025-03-02T05:00:30.476ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1352025-03-02T05:00:30.476ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1362025-03-02T05:00:30.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1372025-03-02T05:00:30.491ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1382025-03-02T05:00:30.495ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1392025-03-02T05:00:30.495ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1402025-03-02T05:00:30.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1412025-03-02T05:00:30.497ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1422025-03-02T05:00:30.498ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1432025-03-02T05:00:30.499ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1442025-03-02T05:00:30.499ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1452025-03-02T05:00:30.501ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1462025-03-02T05:00:30.501ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1472025-03-02T05:00:30.502ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
1482025-03-02T05:00:30.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
1492025-03-02T05:00:30.502ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 60.567939ms iteration = 10 status = {"errors":[],"garbage_collect_requested":[]}
1502025-03-02T05:00:30.508ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1512025-03-02T05:00:30.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1522025-03-02T05:00:30.514ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1532025-03-02T05:00:30.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1542025-03-02T05:00:30.521ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1552025-03-02T05:00:30.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1562025-03-02T05:00:30.524ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1572025-03-02T05:00:30.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1582025-03-02T05:00:30.528ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 86.646218ms iteration = 10 status = {"errors":[],"finish_invoked_ok":[]}
1592025-03-02T05:00:30.529ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1602025-03-02T05:00:30.529ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1612025-03-02T05:00:30.529ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 87.253757ms iteration = 10 status = {"errors":[],"requests_created_ok":[]}
1622025-03-02T05:00:30.529ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1632025-03-02T05:00:30.529ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
1642025-03-02T05:00:30.529ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1652025-03-02T05:00:30.529ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
1662025-03-02T05:00:30.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1672025-03-02T05:00:30.530ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
1682025-03-02T05:00:30.531ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
1692025-03-02T05:00:30.531ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1702025-03-02T05:00:30.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1712025-03-02T05:00:30.531ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
1722025-03-02T05:00:30.532ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1732025-03-02T05:00:30.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1742025-03-02T05:00:30.533ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
1752025-03-02T05:00:30.533ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 91.337309ms iteration = 10 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
1762025-03-02T05:00:30.536ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3359, tv_nsec: 766722400 } })
1772025-03-02T05:00:30.536ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
1782025-03-02T05:00:30.560ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 118.318401ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
1792025-03-02T05:00:30.567ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1802025-03-02T05:00:30.568ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1812025-03-02T05:00:30.568ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1822025-03-02T05:00:30.568ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
1832025-03-02T05:00:30.568ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1842025-03-02T05:00:30.569ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1852025-03-02T05:00:30.569ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1862025-03-02T05:00:30.569ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 127.757584ms iteration = 28 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
1872025-03-02T05:00:30.571ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
1882025-03-02T05:00:30.571ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1892025-03-02T05:00:30.571ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1902025-03-02T05:00:30.572ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
1912025-03-02T05:00:30.572ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1922025-03-02T05:00:30.572ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1932025-03-02T05:00:30.572ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
1942025-03-02T05:00:30.572ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1952025-03-02T05:00:30.572ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
1962025-03-02T05:00:30.572ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1972025-03-02T05:00:30.572ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b1d3ee4-9145-4ae9-8748-86ccb82923d3", "content-length": "26", "date": "Sun, 02 Mar 2025 05:00:30 GMT"} })
1982025-03-02T05:00:30.572ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1992025-03-02T05:00:30.573ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 130.894795ms iteration = 10 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
2002025-03-02T05:00:30.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2012025-03-02T05:00:30.575ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
2022025-03-02T05:00:30.575ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2032025-03-02T05:00:30.575ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
2042025-03-02T05:00:30.575ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 133.397877ms iteration = 10 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
2052025-03-02T05:00:30.576ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 138.025417ms iteration = 10 status = 0
2062025-03-02T05:00:30.585ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2072025-03-02T05:00:30.585ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
2082025-03-02T05:00:30.585ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
2092025-03-02T05:00:30.585ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 143.743573ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
2102025-03-02T05:00:30.586ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "77df8ddb-ee68-4f04-b4e2-8e778c5c7b26", "content-length": "26", "date": "Sun, 02 Mar 2025 05:00:30 GMT"} })
2112025-03-02T05:00:30.586ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
2122025-03-02T05:00:30.587ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
2132025-03-02T05:00:30.587ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2142025-03-02T05:00:30.587ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2152025-03-02T05:00:30.587ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
2162025-03-02T05:00:30.590ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
2172025-03-02T05:00:30.590ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 152.819761ms iteration = 28 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
2182025-03-02T05:00:30.611ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2192025-03-02T05:00:30.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2202025-03-02T05:00:30.613ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 175.264483ms iteration = 11 status = {"physical_disks_added":0}
2212025-03-02T05:00:30.619ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
2222025-03-02T05:00:30.619ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 181.416813ms iteration = 10 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
2232025-03-02T05:00:30.644ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
2242025-03-02T05:00:30.644ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
2252025-03-02T05:00:30.645ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "17d0ffd7-6e80-4d81-8090-a527589610d2", "content-length": "26", "date": "Sun, 02 Mar 2025 05:00:30 GMT"} })
2262025-03-02T05:00:30.645ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
2272025-03-02T05:00:30.647ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
2282025-03-02T05:00:30.712ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
2292025-03-02T05:00:30.712ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dbb3ee6-a10b-4c3b-adfc-357bc52bfd1a", "content-length": "1", "date": "Sun, 02 Mar 2025 05:00:30 GMT"} })
2302025-03-02T05:00:30.714ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
2312025-03-02T05:00:30.714ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
2322025-03-02T05:00:30.720ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
2332025-03-02T05:00:30.720ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
2342025-03-02T05:00:30.720ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f8b0e60-6527-4d35-acf1-4efaba23d352", "content-length": "372", "date": "Sun, 02 Mar 2025 05:00:30 GMT"} })
2352025-03-02T05:00:30.720ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2362025-03-02T05:00:30.721ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2372025-03-02T05:00:30.722ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 286.957758ms iteration = 10 status = null
2382025-03-02T05:00:30.737ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 301.358975ms iteration = 10 status = {}
2392025-03-02T05:00:30.762ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
2402025-03-02T05:00:30.762ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
2412025-03-02T05:00:30.762ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "ec7166c9-bd06-4749-b070-7497c65fb593", "date": "Sun, 02 Mar 2025 05:00:30 GMT"} })
2422025-03-02T05:00:30.762ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2432025-03-02T05:00:30.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2442025-03-02T05:00:30.766ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2452025-03-02T05:00:30.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2462025-03-02T05:00:30.768ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2472025-03-02T05:00:30.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2482025-03-02T05:00:30.772ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
2492025-03-02T05:00:30.772ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
2502025-03-02T05:00:30.772ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "e0987083-e7ad-4ba2-992a-c5024ebf77ae", "date": "Sun, 02 Mar 2025 05:00:30 GMT"} })
2512025-03-02T05:00:30.774ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 331.904773ms iteration = 10 status = {}
2522025-03-02T05:00:30.812ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2532025-03-02T05:00:30.812ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2542025-03-02T05:00:30.814ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2552025-03-02T05:00:30.815ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
2562025-03-02T05:00:30.815ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2572025-03-02T05:00:30.816ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2582025-03-02T05:00:30.817ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2592025-03-02T05:00:30.819ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
2602025-03-02T05:00:30.819ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2612025-03-02T05:00:30.819ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2622025-03-02T05:00:30.824ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2632025-03-02T05:00:30.824ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2642025-03-02T05:00:30.826ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
2652025-03-02T05:00:30.826ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2662025-03-02T05:00:30.826ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2672025-03-02T05:00:30.836ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
2682025-03-02T05:00:30.836ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3a3c812-8855-4382-9c95-127323ed88f0", "content-length": "134", "date": "Sun, 02 Mar 2025 05:00:30 GMT"} })
2692025-03-02T05:00:30.836ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
2702025-03-02T05:00:30.836ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
2712025-03-02T05:00:30.837ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
2722025-03-02T05:00:30.837ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
2732025-03-02T05:00:30.837ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2742025-03-02T05:00:30.837ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2752025-03-02T05:00:30.837ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2762025-03-02T05:00:30.837ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
2772025-03-02T05:00:30.837ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2782025-03-02T05:00:30.837ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
2792025-03-02T05:00:30.837ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
2802025-03-02T05:00:30.837ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2812025-03-02T05:00:30.837ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2822025-03-02T05:00:30.838ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
2832025-03-02T05:00:30.838ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "2159a196-dedb-46c7-9e5b-ef47a490c992", "content-length": "2", "date": "Sun, 02 Mar 2025 05:00:30 GMT"} })
2842025-03-02T05:00:30.838ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
2852025-03-02T05:00:30.839ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "2789f47f-1120-4874-a9f3-0a1282a42f94", "content-length": "52", "date": "Sun, 02 Mar 2025 05:00:30 GMT"} })
2862025-03-02T05:00:30.839ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
2872025-03-02T05:00:30.839ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
2882025-03-02T05:00:30.908ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
2892025-03-02T05:00:30.931ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "68d70b6a-e014-447b-8c92-b8eb4ddfaef2", "date": "Sun, 02 Mar 2025 05:00:30 GMT"} })
2902025-03-02T05:00:30.931ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
2912025-03-02T05:00:30.931ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
2922025-03-02T05:00:30.931ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0d85440-1a55-47e0-9243-0a6e466c3c95", "content-length": "691", "date": "Sun, 02 Mar 2025 05:00:30 GMT"} })
2932025-03-02T05:00:30.931ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
2942025-03-02T05:00:30.931ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
2952025-03-02T05:00:30.931ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
2962025-03-02T05:00:30.931ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "e653606e-94dc-49e4-943b-8f35c5d7d263", "content-length": "682", "date": "Sun, 02 Mar 2025 05:00:30 GMT"} })
2972025-03-02T05:00:30.931ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2982025-03-02T05:00:30.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2992025-03-02T05:00:30.933ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 491.849026ms iteration = 10 status = {}
3002025-03-02T05:00:30.935ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3012025-03-02T05:00:30.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3022025-03-02T05:00:30.938ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3032025-03-02T05:00:30.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3042025-03-02T05:00:30.939ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3052025-03-02T05:00:30.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3062025-03-02T05:00:30.946ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 504.785985ms iteration = 10 status = {}
3072025-03-02T05:00:32.901ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 313 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 22af0fc3-5d00-4a4b-bbb6-28c0c7d52d65 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
3082025-03-02T05:00:40.438ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 29 reason = Timeout
3092025-03-02T05:00:40.438ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3102025-03-02T05:00:40.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3112025-03-02T05:00:40.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3122025-03-02T05:00:40.441ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 29 reason = Timeout
3132025-03-02T05:00:40.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3142025-03-02T05:00:40.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3152025-03-02T05:00:40.443ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
3162025-03-02T05:00:40.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3172025-03-02T05:00:40.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3182025-03-02T05:00:40.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3192025-03-02T05:00:40.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3202025-03-02T05:00:40.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3212025-03-02T05:00:40.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3222025-03-02T05:00:40.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3232025-03-02T05:00:40.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
3242025-03-02T05:00:40.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3252025-03-02T05:00:40.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3262025-03-02T05:00:40.460ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
3272025-03-02T05:00:40.460ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.364598ms iteration = 29 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
3282025-03-02T05:00:40.473ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.802104ms iteration = 29 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
3292025-03-02T05:00:42.901ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 309 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = ff621bea-ca3d-45a4-afd7-e1df79cd3d38 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
3302025-03-02T05:00:50.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 30 reason = Timeout
3312025-03-02T05:00:50.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3322025-03-02T05:00:50.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3332025-03-02T05:00:50.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3342025-03-02T05:00:50.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 30 reason = Timeout
3352025-03-02T05:00:50.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3362025-03-02T05:00:50.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3372025-03-02T05:00:50.443ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
3382025-03-02T05:00:50.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3392025-03-02T05:00:50.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3402025-03-02T05:00:50.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3412025-03-02T05:00:50.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3422025-03-02T05:00:50.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3432025-03-02T05:00:50.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3442025-03-02T05:00:50.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3452025-03-02T05:00:50.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
3462025-03-02T05:00:50.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3472025-03-02T05:00:50.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3482025-03-02T05:00:50.458ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
3492025-03-02T05:00:50.458ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.924216ms iteration = 30 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
3502025-03-02T05:00:50.472ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 30.629819ms iteration = 30 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
3512025-03-02T05:00:52.901ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 269 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 5412dc22-0f86-4eb4-9ba1-50a81fc0bca4 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
3522025-03-02T05:00:57.893ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = f4b503f5-6936-4369-8274-e4c3b6750ddc roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
3532025-03-02T05:00:57.894ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = f4b503f5-6936-4369-8274-e4c3b6750ddc resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
3542025-03-02T05:00:57.896ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2348 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = f4b503f5-6936-4369-8274-e4c3b6750ddc response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
3552025-03-02T05:00:57.896ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = 4d574ad7-4328-4c60-9d87-18ea1cca361f roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
3562025-03-02T05:00:57.896ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = 4d574ad7-4328-4c60-9d87-18ea1cca361f resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
3572025-03-02T05:00:57.898ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 1955 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = 4d574ad7-4328-4c60-9d87-18ea1cca361f response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
3582025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 11 reason = Timeout
3592025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 6 reason = Timeout
3602025-03-02T05:01:00.436ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-02 04:51:00.436119175 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
3612025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 6 reason = Timeout
3622025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
3632025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
3642025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3652025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 11 reason = Timeout
3662025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
3672025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 8 reason = Timeout
3682025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 6 reason = Timeout
3692025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 11 reason = Timeout
3702025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3712025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3722025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 6 reason = Timeout
3732025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
3742025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 6 reason = Timeout
3752025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3762025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 8 reason = Timeout
3772025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 6 reason = Timeout
3782025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
3792025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3802025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3812025-03-02T05:01:00.436ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3bf56dce-8353-46bb-a2f0-982fbd876e0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("742da26a-88eb-42ac-9329-e125fe787858.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 3389, tv_nsec: 667133986 } })
3822025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
3832025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3842025-03-02T05:01:00.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3852025-03-02T05:01:00.436ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4a50e69a-1735-4503-96cb-55c6cc854815.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 3389, tv_nsec: 667234518 } })
3862025-03-02T05:01:00.437ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
3872025-03-02T05:01:00.437ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 876.372\u{b5}s iteration = 6 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
3882025-03-02T05:01:00.437ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
3892025-03-02T05:01:00.437ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 954.742\u{b5}s iteration = 6 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
3902025-03-02T05:01:00.437ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3912025-03-02T05:01:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3922025-03-02T05:01:00.438ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3932025-03-02T05:01:00.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3942025-03-02T05:01:00.438ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-02 04:51:00.436119175 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
3952025-03-02T05:01:00.438ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 2.653585ms iteration = 6 status = {"expiration":"2025-03-02T04:51:00.436119175Z","pruned":[]}
3962025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 11 reason = Timeout
3972025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 11 reason = Timeout
3982025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
3992025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4002025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.332933ms iteration = 11 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
4012025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 6 reason = Timeout
4022025-03-02T05:01:00.439ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
4032025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 3 reason = Timeout
4042025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
4052025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
4062025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 7 reason = Timeout
4072025-03-02T05:01:00.439ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca
4082025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 11.52\u{b5}s iteration = 7 status = {"enabled":false,"target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca"}
4092025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 31 reason = Timeout
4102025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 12 reason = Timeout
4112025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4122025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4132025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4142025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4152025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
4162025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
4172025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4182025-03-02T05:01:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
4192025-03-02T05:01:00.439ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3389, tv_nsec: 670423789 } })
4202025-03-02T05:01:00.440ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4212025-03-02T05:01:00.440ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
4222025-03-02T05:01:00.440ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3389, tv_nsec: 671441963 } })
4232025-03-02T05:01:00.441ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
4242025-03-02T05:01:00.441ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
4252025-03-02T05:01:00.441ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4262025-03-02T05:01:00.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4272025-03-02T05:01:00.441ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4282025-03-02T05:01:00.441ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 11 reason = Timeout
4292025-03-02T05:01:00.441ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4302025-03-02T05:01:00.441ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
4312025-03-02T05:01:00.441ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 11 reason = Timeout
4322025-03-02T05:01:00.441ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 11 reason = Timeout
4332025-03-02T05:01:00.441ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
4342025-03-02T05:01:00.441ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4352025-03-02T05:01:00.441ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 11 reason = Timeout
4362025-03-02T05:01:00.441ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 11 reason = Timeout
4372025-03-02T05:01:00.441ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
4382025-03-02T05:01:00.441ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
4392025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 11 reason = Timeout
4402025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
4412025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 2 reason = Timeout
4422025-03-02T05:01:00.442ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
4432025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
4442025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 6 reason = Timeout
4452025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
4462025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 11 reason = Timeout
4472025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4482025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 11 reason = Timeout
4492025-03-02T05:01:00.442ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
4502025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4512025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 6 reason = Timeout
4522025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
4532025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 6 reason = Timeout
4542025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
4552025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 11 reason = Timeout
4562025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
4572025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
4582025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 11 reason = Timeout
4592025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
4602025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 31 reason = Timeout
4612025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4622025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 11 reason = Timeout
4632025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
4642025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 2 reason = Timeout
4652025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
4662025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
4672025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
4682025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
4692025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
4702025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4712025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
4722025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
4732025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
4742025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
4752025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
4762025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4772025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
4782025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
4792025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
4802025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4812025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
4822025-03-02T05:01:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
4832025-03-02T05:01:00.443ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4842025-03-02T05:01:00.443ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4852025-03-02T05:01:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
4862025-03-02T05:01:00.444ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4872025-03-02T05:01:00.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4882025-03-02T05:01:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4892025-03-02T05:01:00.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
4902025-03-02T05:01:00.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
4912025-03-02T05:01:00.446ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
4922025-03-02T05:01:00.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
4932025-03-02T05:01:00.447ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4942025-03-02T05:01:00.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
4952025-03-02T05:01:00.447ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4962025-03-02T05:01:00.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4972025-03-02T05:01:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
4982025-03-02T05:01:00.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
4992025-03-02T05:01:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5002025-03-02T05:01:00.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
5012025-03-02T05:01:00.449ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5022025-03-02T05:01:00.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5032025-03-02T05:01:00.450ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5042025-03-02T05:01:00.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5052025-03-02T05:01:00.451ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5062025-03-02T05:01:00.453ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
5072025-03-02T05:01:00.453ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
5082025-03-02T05:01:00.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
5092025-03-02T05:01:00.453ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5102025-03-02T05:01:00.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5112025-03-02T05:01:00.454ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 12.708475ms iteration = 6 status = {"errors":[],"found_port_ok":[]}
5122025-03-02T05:01:00.456ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5132025-03-02T05:01:00.456ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
5142025-03-02T05:01:00.456ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
5152025-03-02T05:01:00.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5162025-03-02T05:01:00.456ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5172025-03-02T05:01:00.459ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
5182025-03-02T05:01:00.459ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5192025-03-02T05:01:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5202025-03-02T05:01:00.462ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5212025-03-02T05:01:00.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5222025-03-02T05:01:00.468ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5232025-03-02T05:01:00.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5242025-03-02T05:01:00.474ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5252025-03-02T05:01:00.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5262025-03-02T05:01:00.495ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5272025-03-02T05:01:00.500ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5282025-03-02T05:01:00.501ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
5292025-03-02T05:01:00.501ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
5302025-03-02T05:01:00.501ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
5312025-03-02T05:01:00.501ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5322025-03-02T05:01:00.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5332025-03-02T05:01:00.502ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
5342025-03-02T05:01:00.502ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
5352025-03-02T05:01:00.502ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
5362025-03-02T05:01:00.503ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5372025-03-02T05:01:00.503ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5382025-03-02T05:01:00.503ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5392025-03-02T05:01:00.506ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 64.348198ms iteration = 11 status = {"errors":[],"garbage_collect_requested":[]}
5402025-03-02T05:01:00.507ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
5412025-03-02T05:01:00.507ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5422025-03-02T05:01:00.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5432025-03-02T05:01:00.508ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5442025-03-02T05:01:00.508ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
5452025-03-02T05:01:00.508ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
5462025-03-02T05:01:00.508ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5472025-03-02T05:01:00.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5482025-03-02T05:01:00.508ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5492025-03-02T05:01:00.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5502025-03-02T05:01:00.509ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5512025-03-02T05:01:00.509ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 67.812813ms iteration = 11 status = {"errors":[],"finish_invoked_ok":[]}
5522025-03-02T05:01:00.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5532025-03-02T05:01:00.511ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
5542025-03-02T05:01:00.511ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5552025-03-02T05:01:00.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5562025-03-02T05:01:00.515ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
5572025-03-02T05:01:00.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
5582025-03-02T05:01:00.517ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 74.733544ms iteration = 11 status = {"errors":[],"requests_created_ok":[]}
5592025-03-02T05:01:00.519ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5602025-03-02T05:01:00.523ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5612025-03-02T05:01:00.523ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5622025-03-02T05:01:00.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5632025-03-02T05:01:00.525ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5642025-03-02T05:01:00.527ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
5652025-03-02T05:01:00.527ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5662025-03-02T05:01:00.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5672025-03-02T05:01:00.530ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5682025-03-02T05:01:00.530ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5692025-03-02T05:01:00.531ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
5702025-03-02T05:01:00.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
5712025-03-02T05:01:00.532ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
5722025-03-02T05:01:00.532ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
5732025-03-02T05:01:00.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
5742025-03-02T05:01:00.532ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5752025-03-02T05:01:00.533ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
5762025-03-02T05:01:00.533ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5772025-03-02T05:01:00.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5782025-03-02T05:01:00.535ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
5792025-03-02T05:01:00.535ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
5802025-03-02T05:01:00.535ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 93.184084ms iteration = 6 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
5812025-03-02T05:01:00.535ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5822025-03-02T05:01:00.536ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
5832025-03-02T05:01:00.536ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5842025-03-02T05:01:00.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5852025-03-02T05:01:00.538ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
5862025-03-02T05:01:00.538ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
5872025-03-02T05:01:00.538ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 99.308914ms iteration = 6 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
5882025-03-02T05:01:00.539ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5892025-03-02T05:01:00.539ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5902025-03-02T05:01:00.539ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5912025-03-02T05:01:00.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5922025-03-02T05:01:00.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5932025-03-02T05:01:00.541ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
5942025-03-02T05:01:00.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
5952025-03-02T05:01:00.541ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5962025-03-02T05:01:00.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5972025-03-02T05:01:00.543ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5982025-03-02T05:01:00.544ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5992025-03-02T05:01:00.544ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6002025-03-02T05:01:00.545ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3389, tv_nsec: 775820722 } })
6012025-03-02T05:01:00.545ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
6022025-03-02T05:01:00.546ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
6032025-03-02T05:01:00.546ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
6042025-03-02T05:01:00.547ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
6052025-03-02T05:01:00.557ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6062025-03-02T05:01:00.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6072025-03-02T05:01:00.565ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
6082025-03-02T05:01:00.565ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
6092025-03-02T05:01:00.580ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
6102025-03-02T05:01:00.581ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
6112025-03-02T05:01:00.593ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
6122025-03-02T05:01:00.593ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 151.24489ms iteration = 11 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
6132025-03-02T05:01:00.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
6142025-03-02T05:01:00.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
6152025-03-02T05:01:00.596ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6162025-03-02T05:01:00.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6172025-03-02T05:01:00.602ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 160.758874ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
6182025-03-02T05:01:00.607ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
6192025-03-02T05:01:00.607ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
6202025-03-02T05:01:00.607ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 165.8385ms iteration = 11 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
6212025-03-02T05:01:00.609ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 167.077776ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
6222025-03-02T05:01:00.612ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
6232025-03-02T05:01:00.612ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
6242025-03-02T05:01:00.612ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6252025-03-02T05:01:00.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6262025-03-02T05:01:00.614ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
6272025-03-02T05:01:00.614ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6282025-03-02T05:01:00.614ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2bf2b48-f8b7-4188-998b-6dd548e1455c", "content-length": "26", "date": "Sun, 02 Mar 2025 05:01:00 GMT"} })
6292025-03-02T05:01:00.614ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6302025-03-02T05:01:00.616ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
6312025-03-02T05:01:00.637ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6322025-03-02T05:01:00.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6332025-03-02T05:01:00.685ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
6342025-03-02T05:01:00.685ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6352025-03-02T05:01:00.685ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 245.942733ms iteration = 12 status = {"physical_disks_added":0}
6362025-03-02T05:01:00.685ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea5ad380-c7d6-4f7f-8169-06d6c4f6faae", "content-length": "26", "date": "Sun, 02 Mar 2025 05:01:00 GMT"} })
6372025-03-02T05:01:00.685ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6382025-03-02T05:01:00.686ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
6392025-03-02T05:01:00.700ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
6402025-03-02T05:01:00.738ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
6412025-03-02T05:01:00.738ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
6422025-03-02T05:01:00.739ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "e54719a3-3686-48cf-8298-48c7e72e0045", "content-length": "372", "date": "Sun, 02 Mar 2025 05:01:00 GMT"} })
6432025-03-02T05:01:00.739ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6442025-03-02T05:01:00.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6452025-03-02T05:01:00.739ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
6462025-03-02T05:01:00.741ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6472025-03-02T05:01:00.741ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
6482025-03-02T05:01:00.741ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 0 version = 2
6492025-03-02T05:01:00.741ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6502025-03-02T05:01:00.741ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 299.783875ms iteration = 31 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
6512025-03-02T05:01:00.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6522025-03-02T05:01:00.743ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
6532025-03-02T05:01:00.743ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6542025-03-02T05:01:00.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6552025-03-02T05:01:00.745ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6562025-03-02T05:01:00.745ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6572025-03-02T05:01:00.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6582025-03-02T05:01:00.747ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6592025-03-02T05:01:00.747ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
6602025-03-02T05:01:00.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
6612025-03-02T05:01:00.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
6622025-03-02T05:01:00.748ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
6632025-03-02T05:01:00.748ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6642025-03-02T05:01:00.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6652025-03-02T05:01:00.759ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6662025-03-02T05:01:00.763ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6672025-03-02T05:01:00.763ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
6682025-03-02T05:01:00.763ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6692025-03-02T05:01:00.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6702025-03-02T05:01:00.765ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
6712025-03-02T05:01:00.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
6722025-03-02T05:01:00.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
6732025-03-02T05:01:00.766ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 327.465715ms iteration = 11 status = 0
6742025-03-02T05:01:00.768ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
6752025-03-02T05:01:00.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
6762025-03-02T05:01:00.790ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6772025-03-02T05:01:00.791ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6782025-03-02T05:01:00.792ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
6792025-03-02T05:01:00.792ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6802025-03-02T05:01:00.792ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce299d43-6ce1-412a-ba68-60019d1dd93d", "content-length": "26", "date": "Sun, 02 Mar 2025 05:01:00 GMT"} })
6812025-03-02T05:01:00.792ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6822025-03-02T05:01:00.792ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6832025-03-02T05:01:00.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6842025-03-02T05:01:00.794ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 352.535552ms iteration = 11 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
6852025-03-02T05:01:00.794ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
6862025-03-02T05:01:00.797ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6872025-03-02T05:01:00.798ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6882025-03-02T05:01:00.801ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6892025-03-02T05:01:00.801ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6902025-03-02T05:01:00.801ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
6912025-03-02T05:01:00.801ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6922025-03-02T05:01:00.801ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6932025-03-02T05:01:00.802ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6942025-03-02T05:01:00.805ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
6952025-03-02T05:01:00.805ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
6962025-03-02T05:01:00.805ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-02 04:56:25.487890 UTC dns_group = internal generation = 1
6972025-03-02T05:01:00.805ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 369.201539ms iteration = 6 status = {"generation":1}
6982025-03-02T05:01:00.807ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6992025-03-02T05:01:00.808ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7002025-03-02T05:01:00.808ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7012025-03-02T05:01:00.809ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7022025-03-02T05:01:00.811ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
7032025-03-02T05:01:00.811ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 371.799723ms iteration = 11 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
7042025-03-02T05:01:00.811ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
7052025-03-02T05:01:00.811ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
7062025-03-02T05:01:00.811ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
7072025-03-02T05:01:00.812ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
7082025-03-02T05:01:00.812ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7092025-03-02T05:01:00.812ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7102025-03-02T05:01:00.813ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 1 version = 2
7112025-03-02T05:01:00.813ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
7122025-03-02T05:01:00.813ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-02 04:56:26.100923 UTC dns_group = external generation = 2
7132025-03-02T05:01:00.813ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 376.93593ms iteration = 6 status = {"generation":2}
7142025-03-02T05:01:00.815ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-02T04:55:57.280988Z, time_modified: 2025-03-02T04:55:57.280988Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }) }
7152025-03-02T05:01:00.815ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 378.979976ms iteration = 6 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]}},"default_endpoint":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
7162025-03-02T05:01:00.815ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
7172025-03-02T05:01:00.816ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "56362f4c-aa06-4b59-b950-081f744f093f", "content-length": "1", "date": "Sun, 02 Mar 2025 05:00:59 GMT"} })
7182025-03-02T05:01:00.817ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7192025-03-02T05:01:00.817ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7202025-03-02T05:01:00.818ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7212025-03-02T05:01:00.818ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
7222025-03-02T05:01:00.818ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 379.27336ms iteration = 31 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
7232025-03-02T05:01:00.819ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
7242025-03-02T05:01:00.819ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 383.762579ms iteration = 11 status = null
7252025-03-02T05:01:00.820ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
7262025-03-02T05:01:00.820ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "182abf31-5fa9-4a05-b913-b8c3e4731c68", "date": "Sun, 02 Mar 2025 05:01:00 GMT"} })
7272025-03-02T05:01:00.820ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7282025-03-02T05:01:00.820ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
7292025-03-02T05:01:00.820ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
7302025-03-02T05:01:00.820ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7312025-03-02T05:01:00.820ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
7322025-03-02T05:01:00.821ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7332025-03-02T05:01:00.822ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7342025-03-02T05:01:00.824ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7352025-03-02T05:01:00.824ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7362025-03-02T05:01:00.827ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
7372025-03-02T05:01:00.827ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
7382025-03-02T05:01:00.827ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
7392025-03-02T05:01:00.827ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
7402025-03-02T05:01:00.827ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "0076d769-5af2-438f-a161-e506a786178e", "date": "Sun, 02 Mar 2025 05:01:00 GMT"} })
7412025-03-02T05:01:00.829ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7422025-03-02T05:01:00.829ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 387.068232ms iteration = 11 status = {}
7432025-03-02T05:01:00.830ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
7442025-03-02T05:01:00.830ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
7452025-03-02T05:01:00.830ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
7462025-03-02T05:01:00.831ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7472025-03-02T05:01:00.835ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
7482025-03-02T05:01:00.835ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
7492025-03-02T05:01:00.835ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
7502025-03-02T05:01:00.835ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7512025-03-02T05:01:00.835ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7522025-03-02T05:01:00.836ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7532025-03-02T05:01:00.837ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7542025-03-02T05:01:00.837ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7552025-03-02T05:01:00.839ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
7562025-03-02T05:01:00.839ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
7572025-03-02T05:01:00.839ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
7582025-03-02T05:01:00.843ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
7592025-03-02T05:01:00.843ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
7602025-03-02T05:01:00.845ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7612025-03-02T05:01:00.846ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7622025-03-02T05:01:00.846ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
7632025-03-02T05:01:00.846ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
7642025-03-02T05:01:00.846ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
7652025-03-02T05:01:00.847ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
7662025-03-02T05:01:00.847ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7672025-03-02T05:01:00.847ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7682025-03-02T05:01:00.850ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
7692025-03-02T05:01:00.850ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
7702025-03-02T05:01:00.869ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7712025-03-02T05:01:00.869ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7722025-03-02T05:01:00.884ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
7732025-03-02T05:01:00.884ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 442.115199ms iteration = 6 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
7742025-03-02T05:01:00.896ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 460.130304ms iteration = 11 status = {}
7752025-03-02T05:01:00.974ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
7762025-03-02T05:01:00.992ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
7772025-03-02T05:01:00.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
7782025-03-02T05:01:00.993ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7792025-03-02T05:01:00.994ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7802025-03-02T05:01:00.994ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7812025-03-02T05:01:00.995ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
7822025-03-02T05:01:00.995ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
7832025-03-02T05:01:00.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
7842025-03-02T05:01:00.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
7852025-03-02T05:01:00.997ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
7862025-03-02T05:01:00.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
7872025-03-02T05:01:00.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
7882025-03-02T05:01:00.998ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7892025-03-02T05:01:00.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7902025-03-02T05:01:01.004ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
7912025-03-02T05:01:01.004ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "b88be176-7250-4a16-989d-5575d1d5bbdd", "date": "Sun, 02 Mar 2025 05:01:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7922025-03-02T05:01:01.004ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7932025-03-02T05:01:01.006ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "4167742e-2de9-41e1-b248-3284aecc61fa", "date": "Sun, 02 Mar 2025 05:01:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7942025-03-02T05:01:01.006ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7952025-03-02T05:01:01.008ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7962025-03-02T05:01:01.008ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 572.140522ms iteration = 8 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
7972025-03-02T05:01:01.034ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
7982025-03-02T05:01:01.034ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "480e7f5d-89cb-41d1-8faa-3b74726c6d1b", "content-length": "691", "date": "Sun, 02 Mar 2025 05:01:00 GMT"} })
7992025-03-02T05:01:01.034ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
8002025-03-02T05:01:01.034ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
8012025-03-02T05:01:01.034ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
8022025-03-02T05:01:01.036ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 594.489593ms iteration = 11 status = {}
8032025-03-02T05:01:01.037ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8042025-03-02T05:01:01.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8052025-03-02T05:01:01.039ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
8062025-03-02T05:01:01.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
8072025-03-02T05:01:01.039ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8082025-03-02T05:01:01.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8092025-03-02T05:01:01.041ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8102025-03-02T05:01:01.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8112025-03-02T05:01:01.043ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8122025-03-02T05:01:01.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8132025-03-02T05:01:01.045ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
8142025-03-02T05:01:01.045ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8152025-03-02T05:01:01.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8162025-03-02T05:01:01.046ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "924049b2-b325-49fd-8673-af73cf366732", "content-length": "2", "date": "Sun, 02 Mar 2025 05:01:00 GMT"} })
8172025-03-02T05:01:01.047ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8182025-03-02T05:01:01.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8192025-03-02T05:01:01.051ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:619 sled = http://[fd00:1122:3344:101::1]:12345
8202025-03-02T05:01:01.053ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 611.379794ms iteration = 2 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"operation":"list","target_sled":"fee17c45-a85b-405f-9e58-2ca0ab03dea9","time":"2025-03-02T05:01:01.051764014Z"}]}
8212025-03-02T05:01:01.078ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
8222025-03-02T05:01:01.078ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "8343c972-e4ab-483a-8c1c-0eae4f97aea2", "date": "Sun, 02 Mar 2025 05:01:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
8232025-03-02T05:01:01.078ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
8242025-03-02T05:01:01.079ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8252025-03-02T05:01:01.081ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "35d72a44-3af4-42db-8409-65c8a03a25b7", "date": "Sun, 02 Mar 2025 05:01:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
8262025-03-02T05:01:01.081ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
8272025-03-02T05:01:01.081ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
8282025-03-02T05:01:01.081ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
8292025-03-02T05:01:01.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
8302025-03-02T05:01:01.083ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "3c78dee7-8404-46a1-ad5c-d77ae57fdcba", "date": "Sun, 02 Mar 2025 05:01:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
8312025-03-02T05:01:01.083ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
8322025-03-02T05:01:01.083ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8332025-03-02T05:01:01.084ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
8342025-03-02T05:01:01.085ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
8352025-03-02T05:01:01.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
8362025-03-02T05:01:01.085ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
8372025-03-02T05:01:01.085ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 649.105805ms iteration = 8 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
8382025-03-02T05:01:01.086ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8392025-03-02T05:01:01.088ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
8402025-03-02T05:01:01.090ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 648.65959ms iteration = 3 status = {"blueprint_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","inventory_collection_id":"c9c17d4b-65c0-4a48-afa2-43a0fe16c480","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":4,"num_already_tombstoned":0,"num_inserted":5,"num_not_in_inventory":0,"num_tombstoned":0}}}
8412025-03-02T05:01:01.090ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
8422025-03-02T05:01:01.090ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8432025-03-02T05:01:01.090ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8442025-03-02T05:01:01.092ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8452025-03-02T05:01:01.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8462025-03-02T05:01:01.096ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8472025-03-02T05:01:01.096ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8482025-03-02T05:01:01.099ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8492025-03-02T05:01:01.099ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8502025-03-02T05:01:01.102ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8512025-03-02T05:01:01.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8522025-03-02T05:01:01.103ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8532025-03-02T05:01:01.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8542025-03-02T05:01:01.104ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8552025-03-02T05:01:01.105ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8562025-03-02T05:01:01.105ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8572025-03-02T05:01:01.105ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8582025-03-02T05:01:01.106ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8592025-03-02T05:01:01.107ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
8602025-03-02T05:01:01.107ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8612025-03-02T05:01:01.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8622025-03-02T05:01:01.110ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
8632025-03-02T05:01:01.110ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8642025-03-02T05:01:01.110ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8652025-03-02T05:01:01.111ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8662025-03-02T05:01:01.111ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8672025-03-02T05:01:01.113ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8682025-03-02T05:01:01.113ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8692025-03-02T05:01:01.114ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8702025-03-02T05:01:01.117ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
8712025-03-02T05:01:01.118ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "46c8737e-56c8-497e-8066-fe7a68813216", "content-length": "134", "date": "Sun, 02 Mar 2025 05:01:00 GMT"} })
8722025-03-02T05:01:01.118ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
8732025-03-02T05:01:01.118ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
8742025-03-02T05:01:01.118ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
8752025-03-02T05:01:01.118ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8762025-03-02T05:01:01.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8772025-03-02T05:01:01.118ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
8782025-03-02T05:01:01.118ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
8792025-03-02T05:01:01.118ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
8802025-03-02T05:01:01.118ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
8812025-03-02T05:01:01.118ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
8822025-03-02T05:01:01.118ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
8832025-03-02T05:01:01.118ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
8842025-03-02T05:01:01.118ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
8852025-03-02T05:01:01.118ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
8862025-03-02T05:01:01.118ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8872025-03-02T05:01:01.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8882025-03-02T05:01:01.121ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
8892025-03-02T05:01:01.121ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "15868aa4-bd7f-4d14-89b3-5a2d5f2599d0", "content-length": "2", "date": "Sun, 02 Mar 2025 05:01:00 GMT"} })
8902025-03-02T05:01:01.121ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
8912025-03-02T05:01:01.121ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "acf53293-5c1c-432d-8e83-dca4d5ac87a3", "content-length": "52", "date": "Sun, 02 Mar 2025 05:01:00 GMT"} })
8922025-03-02T05:01:01.121ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
8932025-03-02T05:01:01.121ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
8942025-03-02T05:01:01.124ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8952025-03-02T05:01:01.124ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8962025-03-02T05:01:01.126ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8972025-03-02T05:01:01.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8982025-03-02T05:01:01.128ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8992025-03-02T05:01:01.128ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9002025-03-02T05:01:01.130ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9012025-03-02T05:01:01.130ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9022025-03-02T05:01:01.132ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9032025-03-02T05:01:01.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9042025-03-02T05:01:01.134ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9052025-03-02T05:01:01.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9062025-03-02T05:01:01.135ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9072025-03-02T05:01:01.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9082025-03-02T05:01:01.137ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9092025-03-02T05:01:01.140ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
9102025-03-02T05:01:01.140ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9112025-03-02T05:01:01.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9122025-03-02T05:01:01.141ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9132025-03-02T05:01:01.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9142025-03-02T05:01:01.143ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9152025-03-02T05:01:01.143ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9162025-03-02T05:01:01.144ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9172025-03-02T05:01:01.148ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
9182025-03-02T05:01:01.148ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9192025-03-02T05:01:01.148ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9202025-03-02T05:01:01.154ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 3e4064c5-fe59-43d0-88e0-3ed3e0a5eba7, kind: Service { id: c54e30d8-f5a2-495b-b2d8-a78319addcd7 }, name: Name("external-dns-c54e30d8-f5a2-495b-b2d8-a78319addcd7"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 198, 192])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ef033b7d-0ede-4639-ab7f-6d87d9d01240, kind: Service { id: 4a50e69a-1735-4503-96cb-55c6cc854815 }, name: Name("external-dns-4a50e69a-1735-4503-96cb-55c6cc854815"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 237, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 4d1ce337-a06d-4661-8223-cff03228021a, kind: Service { id: aebed526-5e03-4a58-86b5-1baf17c88392 }, name: Name("nexus-aebed526-5e03-4a58-86b5-1baf17c88392"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 2])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7854a1af-bcd2-41ab-9d60-4bc505686339, kind: Service { id: 091811f5-b683-4991-98f0-82afb4d1f635 }, name: Name("nexus-091811f5-b683-4991-98f0-82afb4d1f635"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b52288c2-a03b-488b-890b-db71040a1e60, kind: Service { id: feddb706-b221-4d1f-920c-1d1979b5460b }, name: Name("nexus-feddb706-b221-4d1f-920c-1d1979b5460b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 242, 53])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
9212025-03-02T05:01:01.154ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9222025-03-02T05:01:01.154ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9232025-03-02T05:01:01.155ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9242025-03-02T05:01:01.157ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
9252025-03-02T05:01:01.157ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9262025-03-02T05:01:01.157ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9272025-03-02T05:01:01.164ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
9282025-03-02T05:01:01.168ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 3e4064c5-fe59-43d0-88e0-3ed3e0a5eba7, kind: Service { id: c54e30d8-f5a2-495b-b2d8-a78319addcd7 }, name: Name("external-dns-c54e30d8-f5a2-495b-b2d8-a78319addcd7"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 198, 192])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ef033b7d-0ede-4639-ab7f-6d87d9d01240, kind: Service { id: 4a50e69a-1735-4503-96cb-55c6cc854815 }, name: Name("external-dns-4a50e69a-1735-4503-96cb-55c6cc854815"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 237, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 4d1ce337-a06d-4661-8223-cff03228021a, kind: Service { id: aebed526-5e03-4a58-86b5-1baf17c88392 }, name: Name("nexus-aebed526-5e03-4a58-86b5-1baf17c88392"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 2])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7854a1af-bcd2-41ab-9d60-4bc505686339, kind: Service { id: 091811f5-b683-4991-98f0-82afb4d1f635 }, name: Name("nexus-091811f5-b683-4991-98f0-82afb4d1f635"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b52288c2-a03b-488b-890b-db71040a1e60, kind: Service { id: feddb706-b221-4d1f-920c-1d1979b5460b }, name: Name("nexus-feddb706-b221-4d1f-920c-1d1979b5460b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 242, 53])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
9292025-03-02T05:01:01.168ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
9302025-03-02T05:01:01.219ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "6f0cedd1-5306-40bb-9a20-3a264b9ffc5c", "date": "Sun, 02 Mar 2025 05:01:01 GMT"} })
9312025-03-02T05:01:01.219ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
9322025-03-02T05:01:01.219ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
9332025-03-02T05:01:01.219ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa8a85a9-acb0-434d-b29d-bf549e79baa1", "content-length": "682", "date": "Sun, 02 Mar 2025 05:01:01 GMT"} })
9342025-03-02T05:01:01.219ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9352025-03-02T05:01:01.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9362025-03-02T05:01:01.222ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9372025-03-02T05:01:01.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9382025-03-02T05:01:01.227ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9392025-03-02T05:01:01.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9402025-03-02T05:01:01.230ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9412025-03-02T05:01:01.230ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9422025-03-02T05:01:01.236ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 795.022566ms iteration = 11 status = {}
9432025-03-02T05:01:01.277ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: fee17c45-a85b-405f-9e58-2ca0ab03dea9, time_created: 2025-03-02T04:56:21.866513Z, time_modified: 2025-03-02T04:56:21.866513Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: d6094382-b327-4998-b53d-b2d98a53005b, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
9442025-03-02T05:01:01.278ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
9452025-03-02T05:01:01.278ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9462025-03-02T05:01:01.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9472025-03-02T05:01:01.280ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9482025-03-02T05:01:01.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9492025-03-02T05:01:01.281ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9502025-03-02T05:01:01.284ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
9512025-03-02T05:01:01.314ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
9522025-03-02T05:01:01.362ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "e0224576-eefb-44ff-bd20-9ef87d0b6f02", "date": "Sun, 02 Mar 2025 05:01:00 GMT"} })
9532025-03-02T05:01:01.364ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
9542025-03-02T05:01:01.364ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 922.560427ms
9552025-03-02T05:01:01.364ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 922.569728ms iteration = 2 status = {}
9562025-03-02T05:01:02.901ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 273 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 31c5f3c0-8b05-4e9b-98d9-997a82cd2de7 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
9572025-03-02T05:01:10.438ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 32 reason = Timeout
9582025-03-02T05:01:10.438ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9592025-03-02T05:01:10.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9602025-03-02T05:01:10.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9612025-03-02T05:01:10.441ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 32 reason = Timeout
9622025-03-02T05:01:10.441ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9632025-03-02T05:01:10.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9642025-03-02T05:01:10.442ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9652025-03-02T05:01:10.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9662025-03-02T05:01:10.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9672025-03-02T05:01:10.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9682025-03-02T05:01:10.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9692025-03-02T05:01:10.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9702025-03-02T05:01:10.451ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
9712025-03-02T05:01:10.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9722025-03-02T05:01:10.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9732025-03-02T05:01:10.464ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
9742025-03-02T05:01:10.464ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.934018ms iteration = 32 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
9752025-03-02T05:01:10.470ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9762025-03-02T05:01:10.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9772025-03-02T05:01:10.505ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.222076ms iteration = 32 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
9782025-03-02T05:01:12.901ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 303 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = e00ab91b-9ae6-476a-ba8f-c0c65cf86678 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
9792025-03-02T05:01:20.438ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 33 reason = Timeout
9802025-03-02T05:01:20.438ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9812025-03-02T05:01:20.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9822025-03-02T05:01:20.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9832025-03-02T05:01:20.441ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 33 reason = Timeout
9842025-03-02T05:01:20.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9852025-03-02T05:01:20.442ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9862025-03-02T05:01:20.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9872025-03-02T05:01:20.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9882025-03-02T05:01:20.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9892025-03-02T05:01:20.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9902025-03-02T05:01:20.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9912025-03-02T05:01:20.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9922025-03-02T05:01:20.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
9932025-03-02T05:01:20.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9942025-03-02T05:01:20.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9952025-03-02T05:01:20.459ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
9962025-03-02T05:01:20.459ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.988103ms iteration = 33 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
9972025-03-02T05:01:20.472ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9982025-03-02T05:01:20.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9992025-03-02T05:01:20.473ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.667182ms iteration = 33 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10002025-03-02T05:01:22.901ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 429 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 0b77d764-fdc2-4b4e-8933-aafaf39b64ce response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
10012025-03-02T05:01:27.894ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = 82a4dbf2-a399-45ee-9500-0f7f5c9cff84 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
10022025-03-02T05:01:27.894ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = 82a4dbf2-a399-45ee-9500-0f7f5c9cff84 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
10032025-03-02T05:01:27.897ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2992 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = 82a4dbf2-a399-45ee-9500-0f7f5c9cff84 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
10042025-03-02T05:01:27.897ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = 849e7aef-a12f-4522-8227-cca6cd34e2a4 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
10052025-03-02T05:01:27.898ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = 849e7aef-a12f-4522-8227-cca6cd34e2a4 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
10062025-03-02T05:01:27.900ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2469 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:65104 req_id = 849e7aef-a12f-4522-8227-cca6cd34e2a4 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
10072025-03-02T05:01:30.435ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 12 reason = Timeout
10082025-03-02T05:01:30.435ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 12 reason = Timeout
10092025-03-02T05:01:30.435ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
10102025-03-02T05:01:30.435ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 12 reason = Timeout
10112025-03-02T05:01:30.435ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
10122025-03-02T05:01:30.435ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
10132025-03-02T05:01:30.435ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
10142025-03-02T05:01:30.438ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3419, tv_nsec: 668437006 } })
10152025-03-02T05:01:30.438ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3419, tv_nsec: 668562228 } })
10162025-03-02T05:01:30.438ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
10172025-03-02T05:01:30.438ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 12 reason = Timeout
10182025-03-02T05:01:30.438ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 34 reason = Timeout
10192025-03-02T05:01:30.438ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10202025-03-02T05:01:30.438ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 13 reason = Timeout
10212025-03-02T05:01:30.438ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10222025-03-02T05:01:30.438ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10232025-03-02T05:01:30.438ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 12 reason = Timeout
10242025-03-02T05:01:30.438ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10252025-03-02T05:01:30.438ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
10262025-03-02T05:01:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10272025-03-02T05:01:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10282025-03-02T05:01:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10292025-03-02T05:01:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10302025-03-02T05:01:30.438ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.375744ms iteration = 12 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
10312025-03-02T05:01:30.440ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10322025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 12 reason = Timeout
10332025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 12 reason = Timeout
10342025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
10352025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10362025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 12 reason = Timeout
10372025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
10382025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
10392025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 12 reason = Timeout
10402025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
10412025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 34 reason = Timeout
10422025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10432025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 12 reason = Timeout
10442025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
10452025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 12 reason = Timeout
10462025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10472025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 12 reason = Timeout
10482025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10492025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 12 reason = Timeout
10502025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
10512025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 12 reason = Timeout
10522025-03-02T05:01:30.441ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
10532025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10542025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 12 reason = Timeout
10552025-03-02T05:01:30.441ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
10562025-03-02T05:01:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10572025-03-02T05:01:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
10582025-03-02T05:01:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10592025-03-02T05:01:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
10602025-03-02T05:01:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10612025-03-02T05:01:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
10622025-03-02T05:01:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
10632025-03-02T05:01:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
10642025-03-02T05:01:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10652025-03-02T05:01:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10662025-03-02T05:01:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
10672025-03-02T05:01:30.442ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 12 reason = Timeout
10682025-03-02T05:01:30.442ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
10692025-03-02T05:01:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
10702025-03-02T05:01:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
10712025-03-02T05:01:30.443ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10722025-03-02T05:01:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10732025-03-02T05:01:30.444ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10742025-03-02T05:01:30.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10752025-03-02T05:01:30.444ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10762025-03-02T05:01:30.444ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10772025-03-02T05:01:30.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
10782025-03-02T05:01:30.447ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10792025-03-02T05:01:30.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
10802025-03-02T05:01:30.448ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10812025-03-02T05:01:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10822025-03-02T05:01:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
10832025-03-02T05:01:30.448ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10842025-03-02T05:01:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10852025-03-02T05:01:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10862025-03-02T05:01:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10872025-03-02T05:01:30.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10882025-03-02T05:01:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10892025-03-02T05:01:30.451ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10902025-03-02T05:01:30.453ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10912025-03-02T05:01:30.453ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10922025-03-02T05:01:30.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10932025-03-02T05:01:30.455ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
10942025-03-02T05:01:30.455ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
10952025-03-02T05:01:30.455ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
10962025-03-02T05:01:30.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
10972025-03-02T05:01:30.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10982025-03-02T05:01:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
10992025-03-02T05:01:30.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
11002025-03-02T05:01:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
11012025-03-02T05:01:30.462ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11022025-03-02T05:01:30.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11032025-03-02T05:01:30.464ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3419, tv_nsec: 695280566 } })
11042025-03-02T05:01:30.465ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
11052025-03-02T05:01:30.465ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
11062025-03-02T05:01:30.465ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11072025-03-02T05:01:30.465ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11082025-03-02T05:01:30.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
11092025-03-02T05:01:30.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11102025-03-02T05:01:30.468ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
11112025-03-02T05:01:30.468ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11122025-03-02T05:01:30.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11132025-03-02T05:01:30.499ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 57.518819ms iteration = 12 status = {"errors":[],"finish_invoked_ok":[]}
11142025-03-02T05:01:30.508ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 66.391045ms iteration = 12 status = {"errors":[],"garbage_collect_requested":[]}
11152025-03-02T05:01:30.509ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11162025-03-02T05:01:30.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11172025-03-02T05:01:30.513ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11182025-03-02T05:01:30.513ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11192025-03-02T05:01:30.515ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11202025-03-02T05:01:30.516ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
11212025-03-02T05:01:30.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
11222025-03-02T05:01:30.518ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
11232025-03-02T05:01:30.518ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
11242025-03-02T05:01:30.518ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
11252025-03-02T05:01:30.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
11262025-03-02T05:01:30.519ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 76.497966ms iteration = 12 status = {"errors":[],"requests_created_ok":[]}
11272025-03-02T05:01:30.519ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
11282025-03-02T05:01:30.519ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11292025-03-02T05:01:30.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11302025-03-02T05:01:30.521ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
11312025-03-02T05:01:30.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
11322025-03-02T05:01:30.521ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11332025-03-02T05:01:30.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11342025-03-02T05:01:30.521ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11352025-03-02T05:01:30.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11362025-03-02T05:01:30.522ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11372025-03-02T05:01:30.523ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11382025-03-02T05:01:30.524ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
11392025-03-02T05:01:30.524ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
11402025-03-02T05:01:30.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
11412025-03-02T05:01:30.524ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
11422025-03-02T05:01:30.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
11432025-03-02T05:01:30.525ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
11442025-03-02T05:01:30.525ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11452025-03-02T05:01:30.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11462025-03-02T05:01:30.526ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
11472025-03-02T05:01:30.526ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 85.078058ms iteration = 12 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
11482025-03-02T05:01:30.565ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
11492025-03-02T05:01:30.565ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11502025-03-02T05:01:30.565ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bd72757-f10c-4bba-97ca-fecc014cd029", "content-length": "26", "date": "Sun, 02 Mar 2025 05:01:30 GMT"} })
11512025-03-02T05:01:30.565ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11522025-03-02T05:01:30.567ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11532025-03-02T05:01:30.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11542025-03-02T05:01:30.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11552025-03-02T05:01:30.673ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
11562025-03-02T05:01:30.673ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11572025-03-02T05:01:30.673ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d30ab356-5e6f-4d91-af7c-a12262e0a79c", "content-length": "26", "date": "Sun, 02 Mar 2025 05:01:30 GMT"} })
11582025-03-02T05:01:30.673ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11592025-03-02T05:01:30.674ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11602025-03-02T05:01:30.687ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
11612025-03-02T05:01:30.688ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "b33f8a6e-436f-403d-b054-18dadd292b6c", "content-length": "1", "date": "Sun, 02 Mar 2025 05:01:30 GMT"} })
11622025-03-02T05:01:30.688ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
11632025-03-02T05:01:30.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11642025-03-02T05:01:30.753ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
11652025-03-02T05:01:30.753ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
11662025-03-02T05:01:30.753ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "12812a2d-186c-4c2d-8e66-b870015868e9", "content-length": "372", "date": "Sun, 02 Mar 2025 05:01:30 GMT"} })
11672025-03-02T05:01:30.754ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11682025-03-02T05:01:30.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11692025-03-02T05:01:30.756ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 318.411838ms iteration = 12 status = 0
11702025-03-02T05:01:30.757ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
11712025-03-02T05:01:30.757ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11722025-03-02T05:01:30.757ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd1e4f20-672f-42ae-a2f5-439054e674fa", "content-length": "26", "date": "Sun, 02 Mar 2025 05:01:30 GMT"} })
11732025-03-02T05:01:30.757ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11742025-03-02T05:01:30.760ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
11752025-03-02T05:01:30.760ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 324.814071ms iteration = 12 status = null
11762025-03-02T05:01:30.761ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 325.603991ms iteration = 12 status = {}
11772025-03-02T05:01:30.786ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
11782025-03-02T05:01:30.786ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 348.162315ms iteration = 34 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
11792025-03-02T05:01:30.792ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
11802025-03-02T05:01:30.792ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
11812025-03-02T05:01:30.792ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 351.259565ms iteration = 12 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
11822025-03-02T05:01:30.794ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
11832025-03-02T05:01:30.795ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
11842025-03-02T05:01:30.795ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 353.576656ms iteration = 34 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
11852025-03-02T05:01:30.800ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 358.309787ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
11862025-03-02T05:01:30.801ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 359.745526ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
11872025-03-02T05:01:30.809ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11882025-03-02T05:01:30.812ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
11892025-03-02T05:01:30.812ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11902025-03-02T05:01:30.812ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11912025-03-02T05:01:30.818ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
11922025-03-02T05:01:30.818ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
11932025-03-02T05:01:30.825ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 383.955921ms iteration = 12 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
11942025-03-02T05:01:30.842ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
11952025-03-02T05:01:30.842ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 404.438238ms iteration = 12 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
11962025-03-02T05:01:30.865ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 427.062293ms iteration = 13 status = {"physical_disks_added":0}
11972025-03-02T05:01:30.868ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11982025-03-02T05:01:30.869ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11992025-03-02T05:01:30.870ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
12002025-03-02T05:01:30.870ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12012025-03-02T05:01:30.871ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12022025-03-02T05:01:30.873ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12032025-03-02T05:01:30.873ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12042025-03-02T05:01:30.874ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12052025-03-02T05:01:30.876ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
12062025-03-02T05:01:30.876ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12072025-03-02T05:01:30.876ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12082025-03-02T05:01:30.881ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
12092025-03-02T05:01:30.881ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "35dbab34-de7b-4c2f-84bc-54779a834fde", "content-length": "134", "date": "Sun, 02 Mar 2025 05:01:30 GMT"} })
12102025-03-02T05:01:30.881ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
12112025-03-02T05:01:30.881ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
12122025-03-02T05:01:30.881ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
12132025-03-02T05:01:30.881ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
12142025-03-02T05:01:30.881ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
12152025-03-02T05:01:30.881ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
12162025-03-02T05:01:30.881ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
12172025-03-02T05:01:30.881ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
12182025-03-02T05:01:30.881ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
12192025-03-02T05:01:30.881ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
12202025-03-02T05:01:30.882ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
12212025-03-02T05:01:30.882ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12222025-03-02T05:01:30.882ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12232025-03-02T05:01:30.892ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
12242025-03-02T05:01:30.892ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7e907ba-2949-42ea-8ea7-8717bf585032", "content-length": "2", "date": "Sun, 02 Mar 2025 05:01:30 GMT"} })
12252025-03-02T05:01:30.892ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
12262025-03-02T05:01:30.892ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1773542-1680-4738-897a-1e0fdf979cd5", "content-length": "52", "date": "Sun, 02 Mar 2025 05:01:30 GMT"} })
12272025-03-02T05:01:30.892ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
12282025-03-02T05:01:30.892ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
12292025-03-02T05:01:30.911ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "904d08fe-c684-4862-b00f-4eba0283516c", "date": "Sun, 02 Mar 2025 05:01:30 GMT"} })
12302025-03-02T05:01:30.911ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12312025-03-02T05:01:30.912ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12322025-03-02T05:01:30.914ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12332025-03-02T05:01:30.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12342025-03-02T05:01:30.916ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12352025-03-02T05:01:30.918ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12362025-03-02T05:01:30.918ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
12372025-03-02T05:01:30.918ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12382025-03-02T05:01:30.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12392025-03-02T05:01:30.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12402025-03-02T05:01:30.922ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
12412025-03-02T05:01:30.922ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
12422025-03-02T05:01:30.922ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "684a9898-0908-456b-abc6-e1038209d42b", "date": "Sun, 02 Mar 2025 05:01:30 GMT"} })
12432025-03-02T05:01:30.924ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 483.102153ms iteration = 12 status = {}
12442025-03-02T05:01:30.996ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "69af9d6e-1d77-4151-8beb-fb4d64c21912", "date": "Sun, 02 Mar 2025 05:01:30 GMT"} })
12452025-03-02T05:01:30.996ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
12462025-03-02T05:01:30.996ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
12472025-03-02T05:01:30.997ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "8246e6bd-0b5c-4929-9569-bf92548af6cb", "content-length": "682", "date": "Sun, 02 Mar 2025 05:01:30 GMT"} })
12482025-03-02T05:01:30.997ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12492025-03-02T05:01:30.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12502025-03-02T05:01:31.000ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12512025-03-02T05:01:31.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12522025-03-02T05:01:31.003ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12532025-03-02T05:01:31.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12542025-03-02T05:01:31.005ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12552025-03-02T05:01:31.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12562025-03-02T05:01:31.009ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
12572025-03-02T05:01:31.010ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "82ba8756-cb98-40ea-97b2-d0ef4a8d1eb1", "content-length": "691", "date": "Sun, 02 Mar 2025 05:01:30 GMT"} })
12582025-03-02T05:01:31.010ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
12592025-03-02T05:01:31.010ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
12602025-03-02T05:01:31.010ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
12612025-03-02T05:01:31.010ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 569.071823ms iteration = 12 status = {}
12622025-03-02T05:01:31.016ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 574.680156ms iteration = 12 status = {}
12632025-03-02T05:01:32.901ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 262 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = e3a0b2e3-e99e-4890-bc3c-1b3e65d95650 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
12642025-03-02T05:01:40.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 35 reason = Timeout
12652025-03-02T05:01:40.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12662025-03-02T05:01:40.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12672025-03-02T05:01:40.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12682025-03-02T05:01:40.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 35 reason = Timeout
12692025-03-02T05:01:40.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12702025-03-02T05:01:40.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12712025-03-02T05:01:40.443ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12722025-03-02T05:01:40.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12732025-03-02T05:01:40.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12742025-03-02T05:01:40.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12752025-03-02T05:01:40.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12762025-03-02T05:01:40.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12772025-03-02T05:01:40.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12782025-03-02T05:01:40.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12792025-03-02T05:01:40.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
12802025-03-02T05:01:40.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12812025-03-02T05:01:40.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12822025-03-02T05:01:40.461ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
12832025-03-02T05:01:40.461ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.409072ms iteration = 35 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
12842025-03-02T05:01:40.473ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.388089ms iteration = 35 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
12852025-03-02T05:01:42.901ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 292 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 188a90d0-d9b5-4691-aa73-03c81dde9dba response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
12862025-03-02T05:01:50.438ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 36 reason = Timeout
12872025-03-02T05:01:50.438ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12882025-03-02T05:01:50.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12892025-03-02T05:01:50.441ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 36 reason = Timeout
12902025-03-02T05:01:50.441ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12912025-03-02T05:01:50.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12922025-03-02T05:01:50.469ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12932025-03-02T05:01:50.471ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12942025-03-02T05:01:50.471ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12952025-03-02T05:01:50.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12962025-03-02T05:01:50.472ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12972025-03-02T05:01:50.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12982025-03-02T05:01:50.474ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12992025-03-02T05:01:50.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13002025-03-02T05:01:50.477ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13012025-03-02T05:01:50.479ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
13022025-03-02T05:01:50.480ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13032025-03-02T05:01:50.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13042025-03-02T05:01:50.492ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
13052025-03-02T05:01:50.492ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 53.401206ms iteration = 36 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
13062025-03-02T05:01:50.504ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.668917ms iteration = 36 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
13072025-03-02T05:01:52.901ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 417 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = db78c798-35d9-4998-95ef-e2cae060f747 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
13082025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 13 reason = Timeout
13092025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 7 reason = Timeout
13102025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 7 reason = Timeout
13112025-03-02T05:02:00.436ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-02 04:52:00.436375602 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
13122025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 7 reason = Timeout
13132025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
13142025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
13152025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13162025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13172025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 7 reason = Timeout
13182025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 9 reason = Timeout
13192025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 7 reason = Timeout
13202025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 13 reason = Timeout
13212025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 7 reason = Timeout
13222025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
13232025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
13242025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13252025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 13 reason = Timeout
13262025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 9 reason = Timeout
13272025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
13282025-03-02T05:02:00.436ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3bf56dce-8353-46bb-a2f0-982fbd876e0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("742da26a-88eb-42ac-9329-e125fe787858.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 3449, tv_nsec: 667010033 } })
13292025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13302025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13312025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13322025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
13332025-03-02T05:02:00.436ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4a50e69a-1735-4503-96cb-55c6cc854815.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 3449, tv_nsec: 667107714 } })
13342025-03-02T05:02:00.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
13352025-03-02T05:02:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
13362025-03-02T05:02:00.437ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
13372025-03-02T05:02:00.437ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 903.751\u{b5}s iteration = 7 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
13382025-03-02T05:02:00.437ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
13392025-03-02T05:02:00.437ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 894.332\u{b5}s iteration = 7 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
13402025-03-02T05:02:00.437ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13412025-03-02T05:02:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13422025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13432025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13442025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 14 reason = Timeout
13452025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13462025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 13 reason = Timeout
13472025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 8 reason = Timeout
13482025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13492025-03-02T05:02:00.438ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca
13502025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 4 reason = Timeout
13512025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.24\u{b5}s iteration = 8 status = {"enabled":false,"target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca"}
13522025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 37 reason = Timeout
13532025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 7 reason = Timeout
13542025-03-02T05:02:00.438ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
13552025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
13562025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13572025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
13582025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 13 reason = Timeout
13592025-03-02T05:02:00.438ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
13602025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13612025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13622025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
13632025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
13642025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
13652025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13662025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13672025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
13682025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
13692025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
13702025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
13712025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
13722025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
13732025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
13742025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.423794ms iteration = 13 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
13752025-03-02T05:02:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
13762025-03-02T05:02:00.440ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-02 04:52:00.436375602 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
13772025-03-02T05:02:00.440ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.8281ms iteration = 7 status = {"expiration":"2025-03-02T04:52:00.436375602Z","pruned":[]}
13782025-03-02T05:02:00.440ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3449, tv_nsec: 670928074 } })
13792025-03-02T05:02:00.440ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
13802025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13812025-03-02T05:02:00.441ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
13822025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13832025-03-02T05:02:00.441ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3449, tv_nsec: 671582062 } })
13842025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 13 reason = Timeout
13852025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 7 reason = Timeout
13862025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
13872025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
13882025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 13 reason = Timeout
13892025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
13902025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 13 reason = Timeout
13912025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
13922025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 13 reason = Timeout
13932025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13942025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 13 reason = Timeout
13952025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
13962025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 13 reason = Timeout
13972025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
13982025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
13992025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 7 reason = Timeout
14002025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
14012025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 13 reason = Timeout
14022025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 37 reason = Timeout
14032025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
14042025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14052025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 13 reason = Timeout
14062025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
14072025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 13 reason = Timeout
14082025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14092025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 13 reason = Timeout
14102025-03-02T05:02:00.441ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
14112025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14122025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 7 reason = Timeout
14132025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
14142025-03-02T05:02:00.441ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
14152025-03-02T05:02:00.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
14162025-03-02T05:02:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
14172025-03-02T05:02:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
14182025-03-02T05:02:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
14192025-03-02T05:02:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
14202025-03-02T05:02:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14212025-03-02T05:02:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
14222025-03-02T05:02:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
14232025-03-02T05:02:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14242025-03-02T05:02:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14252025-03-02T05:02:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14262025-03-02T05:02:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
14272025-03-02T05:02:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
14282025-03-02T05:02:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
14292025-03-02T05:02:00.443ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 13 reason = Timeout
14302025-03-02T05:02:00.443ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
14312025-03-02T05:02:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
14322025-03-02T05:02:00.443ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14332025-03-02T05:02:00.443ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14342025-03-02T05:02:00.444ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14352025-03-02T05:02:00.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
14362025-03-02T05:02:00.446ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
14372025-03-02T05:02:00.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
14382025-03-02T05:02:00.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
14392025-03-02T05:02:00.446ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14402025-03-02T05:02:00.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14412025-03-02T05:02:00.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
14422025-03-02T05:02:00.446ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
14432025-03-02T05:02:00.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
14442025-03-02T05:02:00.449ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14452025-03-02T05:02:00.450ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
14462025-03-02T05:02:00.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
14472025-03-02T05:02:00.450ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
14482025-03-02T05:02:00.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
14492025-03-02T05:02:00.450ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14502025-03-02T05:02:00.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
14512025-03-02T05:02:00.452ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
14522025-03-02T05:02:00.452ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
14532025-03-02T05:02:00.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
14542025-03-02T05:02:00.459ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 17.717101ms iteration = 7 status = {"errors":[],"found_port_ok":[]}
14552025-03-02T05:02:00.460ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14562025-03-02T05:02:00.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
14572025-03-02T05:02:00.462ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14582025-03-02T05:02:00.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14592025-03-02T05:02:00.464ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14602025-03-02T05:02:00.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
14612025-03-02T05:02:00.467ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
14622025-03-02T05:02:00.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
14632025-03-02T05:02:00.495ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14642025-03-02T05:02:00.498ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
14652025-03-02T05:02:00.498ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
14662025-03-02T05:02:00.498ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
14672025-03-02T05:02:00.498ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14682025-03-02T05:02:00.500ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14692025-03-02T05:02:00.503ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
14702025-03-02T05:02:00.503ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14712025-03-02T05:02:00.503ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14722025-03-02T05:02:00.506ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
14732025-03-02T05:02:00.508ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14742025-03-02T05:02:00.509ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
14752025-03-02T05:02:00.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14762025-03-02T05:02:00.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
14772025-03-02T05:02:00.510ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14782025-03-02T05:02:00.511ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14792025-03-02T05:02:00.513ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
14802025-03-02T05:02:00.513ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
14812025-03-02T05:02:00.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
14822025-03-02T05:02:00.514ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14832025-03-02T05:02:00.515ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
14842025-03-02T05:02:00.517ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
14852025-03-02T05:02:00.517ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
14862025-03-02T05:02:00.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
14872025-03-02T05:02:00.521ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14882025-03-02T05:02:00.523ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
14892025-03-02T05:02:00.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
14902025-03-02T05:02:00.524ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
14912025-03-02T05:02:00.524ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
14922025-03-02T05:02:00.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
14932025-03-02T05:02:00.527ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14942025-03-02T05:02:00.531ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
14952025-03-02T05:02:00.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
14962025-03-02T05:02:00.531ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
14972025-03-02T05:02:00.531ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
14982025-03-02T05:02:00.532ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3449, tv_nsec: 762199773 } })
14992025-03-02T05:02:00.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
15002025-03-02T05:02:00.532ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
15012025-03-02T05:02:00.534ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 92.535675ms iteration = 13 status = {"errors":[],"finish_invoked_ok":[]}
15022025-03-02T05:02:00.539ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15032025-03-02T05:02:00.541ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15042025-03-02T05:02:00.542ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
15052025-03-02T05:02:00.542ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15062025-03-02T05:02:00.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15072025-03-02T05:02:00.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15082025-03-02T05:02:00.542ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15092025-03-02T05:02:00.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15102025-03-02T05:02:00.548ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
15112025-03-02T05:02:00.548ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
15122025-03-02T05:02:00.548ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
15132025-03-02T05:02:00.548ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15142025-03-02T05:02:00.548ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
15152025-03-02T05:02:00.549ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 107.851734ms iteration = 13 status = {"errors":[],"garbage_collect_requested":[]}
15162025-03-02T05:02:00.549ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
15172025-03-02T05:02:00.550ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
15182025-03-02T05:02:00.551ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 112.791419ms iteration = 4 status = {"nsuccess":5}
15192025-03-02T05:02:00.551ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
15202025-03-02T05:02:00.551ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
15212025-03-02T05:02:00.552ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
15222025-03-02T05:02:00.552ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15232025-03-02T05:02:00.552ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
15242025-03-02T05:02:00.552ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
15252025-03-02T05:02:00.552ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15262025-03-02T05:02:00.552ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 110.463489ms iteration = 7 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
15272025-03-02T05:02:00.555ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
15282025-03-02T05:02:00.555ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15292025-03-02T05:02:00.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15302025-03-02T05:02:00.557ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
15312025-03-02T05:02:00.557ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15322025-03-02T05:02:00.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15332025-03-02T05:02:00.559ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 117.417179ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
15342025-03-02T05:02:00.570ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 127.475991ms iteration = 13 status = {"errors":[],"requests_created_ok":[]}
15352025-03-02T05:02:00.571ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
15362025-03-02T05:02:00.571ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
15372025-03-02T05:02:00.571ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
15382025-03-02T05:02:00.571ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
15392025-03-02T05:02:00.572ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
15402025-03-02T05:02:00.575ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15412025-03-02T05:02:00.575ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
15422025-03-02T05:02:00.575ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
15432025-03-02T05:02:00.575ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
15442025-03-02T05:02:00.575ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
15452025-03-02T05:02:00.575ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
15462025-03-02T05:02:00.576ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
15472025-03-02T05:02:00.576ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
15482025-03-02T05:02:00.577ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
15492025-03-02T05:02:00.577ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
15502025-03-02T05:02:00.577ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 138.777417ms iteration = 7 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
15512025-03-02T05:02:00.579ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
15522025-03-02T05:02:00.579ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
15532025-03-02T05:02:00.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
15542025-03-02T05:02:00.579ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15552025-03-02T05:02:00.579ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
15562025-03-02T05:02:00.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15572025-03-02T05:02:00.579ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 140.919346ms iteration = 37 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
15582025-03-02T05:02:00.581ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15592025-03-02T05:02:00.581ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15602025-03-02T05:02:00.583ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15612025-03-02T05:02:00.583ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
15622025-03-02T05:02:00.583ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
15632025-03-02T05:02:00.583ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
15642025-03-02T05:02:00.583ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15652025-03-02T05:02:00.584ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
15662025-03-02T05:02:00.584ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
15672025-03-02T05:02:00.584ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
15682025-03-02T05:02:00.584ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
15692025-03-02T05:02:00.584ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
15702025-03-02T05:02:00.584ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
15712025-03-02T05:02:00.585ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
15722025-03-02T05:02:00.585ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
15732025-03-02T05:02:00.585ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
15742025-03-02T05:02:00.585ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15752025-03-02T05:02:00.586ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15762025-03-02T05:02:00.587ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15772025-03-02T05:02:00.588ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
15782025-03-02T05:02:00.588ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
15792025-03-02T05:02:00.588ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1d80ee4-20a3-40d8-861f-29a52af511cb", "content-length": "26", "date": "Sun, 02 Mar 2025 05:02:00 GMT"} })
15802025-03-02T05:02:00.588ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
15812025-03-02T05:02:00.589ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
15822025-03-02T05:02:00.589ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
15832025-03-02T05:02:00.589ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
15842025-03-02T05:02:00.590ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
15852025-03-02T05:02:00.590ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
15862025-03-02T05:02:00.590ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15872025-03-02T05:02:00.590ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
15882025-03-02T05:02:00.590ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
15892025-03-02T05:02:00.590ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15902025-03-02T05:02:00.590ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
15912025-03-02T05:02:00.592ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
15922025-03-02T05:02:00.592ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 0 version = 2
15932025-03-02T05:02:00.592ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
15942025-03-02T05:02:00.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
15952025-03-02T05:02:00.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
15962025-03-02T05:02:00.593ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 154.997579ms iteration = 13 status = 0
15972025-03-02T05:02:00.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
15982025-03-02T05:02:00.594ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
15992025-03-02T05:02:00.594ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
16002025-03-02T05:02:00.594ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-02 04:56:25.487890 UTC dns_group = internal generation = 1
16012025-03-02T05:02:00.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 157.857816ms iteration = 7 status = {"generation":1}
16022025-03-02T05:02:00.594ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16032025-03-02T05:02:00.595ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 153.775003ms iteration = 13 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
16042025-03-02T05:02:00.595ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 153.720592ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
16052025-03-02T05:02:00.595ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
16062025-03-02T05:02:00.595ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 154.245829ms iteration = 13 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
16072025-03-02T05:02:00.597ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
16082025-03-02T05:02:00.597ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
16092025-03-02T05:02:00.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
16102025-03-02T05:02:00.599ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
16112025-03-02T05:02:00.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
16122025-03-02T05:02:00.599ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 1 version = 2
16132025-03-02T05:02:00.599ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
16142025-03-02T05:02:00.599ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-02 04:56:26.100923 UTC dns_group = external generation = 2
16152025-03-02T05:02:00.599ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 163.366708ms iteration = 7 status = {"generation":2}
16162025-03-02T05:02:00.602ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 160.49177ms iteration = 37 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
16172025-03-02T05:02:00.607ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16182025-03-02T05:02:00.607ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
16192025-03-02T05:02:00.607ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
16202025-03-02T05:02:00.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
16212025-03-02T05:02:00.621ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
16222025-03-02T05:02:00.621ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 182.873842ms iteration = 13 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
16232025-03-02T05:02:00.639ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-02T04:55:57.280988Z, time_modified: 2025-03-02T04:55:57.280988Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }) }
16242025-03-02T05:02:00.639ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 202.958314ms iteration = 7 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]}},"default_endpoint":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
16252025-03-02T05:02:00.642ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16262025-03-02T05:02:00.642ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16272025-03-02T05:02:00.646ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16282025-03-02T05:02:00.646ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
16292025-03-02T05:02:00.646ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 205.181083ms iteration = 13 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
16302025-03-02T05:02:00.671ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 233.072996ms iteration = 14 status = {"physical_disks_added":0}
16312025-03-02T05:02:00.713ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
16322025-03-02T05:02:00.713ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 271.594868ms iteration = 7 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
16332025-03-02T05:02:00.725ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
16342025-03-02T05:02:00.725ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16352025-03-02T05:02:00.725ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfb174db-7cd7-49d3-9296-bdb705362fbf", "content-length": "26", "date": "Sun, 02 Mar 2025 05:02:00 GMT"} })
16362025-03-02T05:02:00.725ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16372025-03-02T05:02:00.726ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
16382025-03-02T05:02:00.755ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16392025-03-02T05:02:00.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16402025-03-02T05:02:00.757ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16412025-03-02T05:02:00.757ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 321.119873ms iteration = 13 status = {}
16422025-03-02T05:02:00.758ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
16432025-03-02T05:02:00.758ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16442025-03-02T05:02:00.759ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16452025-03-02T05:02:00.760ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16462025-03-02T05:02:00.762ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
16472025-03-02T05:02:00.762ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16482025-03-02T05:02:00.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16492025-03-02T05:02:00.808ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
16502025-03-02T05:02:00.830ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
16512025-03-02T05:02:00.830ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16522025-03-02T05:02:00.830ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3a6b2d1-e160-4ee6-907e-fb4bc4872b57", "content-length": "26", "date": "Sun, 02 Mar 2025 05:02:00 GMT"} })
16532025-03-02T05:02:00.830ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16542025-03-02T05:02:00.832ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
16552025-03-02T05:02:00.841ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
16562025-03-02T05:02:00.841ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
16572025-03-02T05:02:00.841ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "15162ca8-92af-4476-8754-dbecfd011c25", "content-length": "372", "date": "Sun, 02 Mar 2025 05:02:00 GMT"} })
16582025-03-02T05:02:00.841ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16592025-03-02T05:02:00.841ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16602025-03-02T05:02:00.855ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
16612025-03-02T05:02:00.884ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
16622025-03-02T05:02:00.884ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
16632025-03-02T05:02:00.884ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "dd4cb75f-a021-4080-a9a9-d8901e36442d", "date": "Sun, 02 Mar 2025 05:02:00 GMT"} })
16642025-03-02T05:02:00.884ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16652025-03-02T05:02:00.884ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16662025-03-02T05:02:00.886ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16672025-03-02T05:02:00.886ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16682025-03-02T05:02:00.890ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16692025-03-02T05:02:00.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16702025-03-02T05:02:00.965ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
16712025-03-02T05:02:00.965ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
16722025-03-02T05:02:01.014ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
16732025-03-02T05:02:01.014ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "3419c2df-0bbe-41b5-b687-528a9102c38f", "content-length": "1", "date": "Sun, 02 Mar 2025 05:02:00 GMT"} })
16742025-03-02T05:02:01.015ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
16752025-03-02T05:02:01.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
16762025-03-02T05:02:01.082ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
16772025-03-02T05:02:01.083ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7a8c417-0301-4373-8b33-62e3b5a7c182", "content-length": "691", "date": "Sun, 02 Mar 2025 05:02:00 GMT"} })
16782025-03-02T05:02:01.083ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
16792025-03-02T05:02:01.083ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
16802025-03-02T05:02:01.083ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
16812025-03-02T05:02:01.084ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 642.811173ms iteration = 13 status = {}
16822025-03-02T05:02:01.084ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 648.197453ms iteration = 13 status = null
16832025-03-02T05:02:01.085ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
16842025-03-02T05:02:01.085ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
16852025-03-02T05:02:01.115ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
16862025-03-02T05:02:01.122ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
16872025-03-02T05:02:01.122ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:37610
16882025-03-02T05:02:01.122ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:37610 req_id = 1f2367be-0ee4-47b4-84a2-d9060dbf6d47 roles = RoleSet { roles: {} } uri = /metrics/producers
16892025-03-02T05:02:01.123ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:37610 req_id = 1f2367be-0ee4-47b4-84a2-d9060dbf6d47 resource = Database result = Ok(()) uri = /metrics/producers
16902025-03-02T05:02:01.142ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
16912025-03-02T05:02:01.142ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "c15e2704-e1ee-4983-a266-82bb20f89396", "date": "Sun, 02 Mar 2025 05:02:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
16922025-03-02T05:02:01.142ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
16932025-03-02T05:02:01.143ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "e5138ed3-d9da-4bc1-bfb9-4b638efba7e2", "date": "Sun, 02 Mar 2025 05:02:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
16942025-03-02T05:02:01.143ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
16952025-03-02T05:02:01.145ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
16962025-03-02T05:02:01.145ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 708.641431ms iteration = 9 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
16972025-03-02T05:02:01.178ZINFOnexus: assigned collector to new producer collector_id = 8aaa536c-c534-4dd2-aad0-92d8934e5eba file = nexus/src/app/oximeter.rs:76 producer_id = aebed526-5e03-4a58-86b5-1baf17c88392
16982025-03-02T05:02:01.178ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 55437 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:37610 req_id = 1f2367be-0ee4-47b4-84a2-d9060dbf6d47 response_code = 201 uri = /metrics/producers
16992025-03-02T05:02:01.178ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "1f2367be-0ee4-47b4-84a2-d9060dbf6d47", "content-length": "41", "date": "Sun, 02 Mar 2025 05:02:01 GMT"} })
17002025-03-02T05:02:01.179ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "4ba0d306-384f-4581-820e-f4657ce41b96", "date": "Sun, 02 Mar 2025 05:02:01 GMT"} })
17012025-03-02T05:02:01.180ZDEBGnexus (producer-registration-task): registered with nexus successfully
17022025-03-02T05:02:01.180ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
17032025-03-02T05:02:01.181ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 740.004249ms iteration = 13 status = {}
17042025-03-02T05:02:01.197ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
17052025-03-02T05:02:01.198ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "adac95bd-87f7-4690-87f0-896634246018", "date": "Sun, 02 Mar 2025 05:02:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
17062025-03-02T05:02:01.198ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
17072025-03-02T05:02:01.200ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "c344a9ed-9d61-4ca7-b7ed-81e913c2362b", "date": "Sun, 02 Mar 2025 05:02:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
17082025-03-02T05:02:01.200ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
17092025-03-02T05:02:01.202ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "e00a7820-fca6-48ff-bf40-4ad053398e3a", "date": "Sun, 02 Mar 2025 05:02:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
17102025-03-02T05:02:01.202ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
17112025-03-02T05:02:01.204ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
17122025-03-02T05:02:01.204ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 767.70067ms iteration = 9 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
17132025-03-02T05:02:01.211ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17142025-03-02T05:02:01.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17152025-03-02T05:02:01.214ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
17162025-03-02T05:02:01.214ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17172025-03-02T05:02:01.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17182025-03-02T05:02:01.223ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
17192025-03-02T05:02:01.224ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "04dbf3ae-30e6-494d-8a00-9b7f5c3df412", "content-length": "134", "date": "Sun, 02 Mar 2025 05:02:00 GMT"} })
17202025-03-02T05:02:01.224ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
17212025-03-02T05:02:01.224ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
17222025-03-02T05:02:01.224ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
17232025-03-02T05:02:01.224ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
17242025-03-02T05:02:01.224ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17252025-03-02T05:02:01.224ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17262025-03-02T05:02:01.224ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17272025-03-02T05:02:01.224ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
17282025-03-02T05:02:01.224ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17292025-03-02T05:02:01.224ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
17302025-03-02T05:02:01.224ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
17312025-03-02T05:02:01.224ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17322025-03-02T05:02:01.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17332025-03-02T05:02:01.227ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
17342025-03-02T05:02:01.227ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fdc6eb7-6177-4d86-9c02-20aa4ccabcb0", "content-length": "2", "date": "Sun, 02 Mar 2025 05:02:00 GMT"} })
17352025-03-02T05:02:01.227ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
17362025-03-02T05:02:01.227ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ce06a18-ca17-4ce6-bd37-76071f5cf90b", "content-length": "52", "date": "Sun, 02 Mar 2025 05:02:00 GMT"} })
17372025-03-02T05:02:01.227ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
17382025-03-02T05:02:01.227ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
17392025-03-02T05:02:01.310ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "6da2e636-9376-4fec-84a2-c0f61834eb10", "date": "Sun, 02 Mar 2025 05:02:01 GMT"} })
17402025-03-02T05:02:01.310ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
17412025-03-02T05:02:01.310ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
17422025-03-02T05:02:01.310ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "2677a6fa-3a02-4f5e-abbf-1df0fe2e484b", "content-length": "682", "date": "Sun, 02 Mar 2025 05:02:01 GMT"} })
17432025-03-02T05:02:01.310ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17442025-03-02T05:02:01.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17452025-03-02T05:02:01.314ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17462025-03-02T05:02:01.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17472025-03-02T05:02:01.317ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17482025-03-02T05:02:01.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17492025-03-02T05:02:01.319ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17502025-03-02T05:02:01.319ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17512025-03-02T05:02:01.325ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 883.724851ms iteration = 13 status = {}
17522025-03-02T05:02:02.901ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 265 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 0eda28f7-b5b4-4773-91c6-5fea86dd9293 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
17532025-03-02T05:02:10.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 38 reason = Timeout
17542025-03-02T05:02:10.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17552025-03-02T05:02:10.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17562025-03-02T05:02:10.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17572025-03-02T05:02:10.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 38 reason = Timeout
17582025-03-02T05:02:10.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17592025-03-02T05:02:10.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17602025-03-02T05:02:10.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
17612025-03-02T05:02:10.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17622025-03-02T05:02:10.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17632025-03-02T05:02:10.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17642025-03-02T05:02:10.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17652025-03-02T05:02:10.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17662025-03-02T05:02:10.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
17672025-03-02T05:02:10.450ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17682025-03-02T05:02:10.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17692025-03-02T05:02:10.461ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
17702025-03-02T05:02:10.461ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.364292ms iteration = 38 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
17712025-03-02T05:02:10.472ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17722025-03-02T05:02:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17732025-03-02T05:02:10.502ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.806129ms iteration = 38 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
17742025-03-02T05:02:12.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 387 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 2028b9b9-dd09-4496-9e2c-9fc7f552d9c2 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
17752025-03-02T05:02:20.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 39 reason = Timeout
17762025-03-02T05:02:20.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17772025-03-02T05:02:20.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17782025-03-02T05:02:20.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17792025-03-02T05:02:20.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 39 reason = Timeout
17802025-03-02T05:02:20.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17812025-03-02T05:02:20.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17822025-03-02T05:02:20.443ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
17832025-03-02T05:02:20.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17842025-03-02T05:02:20.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17852025-03-02T05:02:20.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17862025-03-02T05:02:20.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17872025-03-02T05:02:20.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17882025-03-02T05:02:20.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
17892025-03-02T05:02:20.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17902025-03-02T05:02:20.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17912025-03-02T05:02:20.461ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
17922025-03-02T05:02:20.461ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.288581ms iteration = 39 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
17932025-03-02T05:02:20.471ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17942025-03-02T05:02:20.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17952025-03-02T05:02:20.473ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.168195ms iteration = 39 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
17962025-03-02T05:02:22.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 282 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = cc245a3d-e86f-480f-8b8f-2f0f26cf55c6 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
17972025-03-02T05:02:30.436ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
17982025-03-02T05:02:30.436ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
17992025-03-02T05:02:30.436ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
18002025-03-02T05:02:30.436ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
18012025-03-02T05:02:30.436ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
18022025-03-02T05:02:30.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
18032025-03-02T05:02:30.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
18042025-03-02T05:02:30.438ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 15 reason = Timeout
18052025-03-02T05:02:30.438ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
18062025-03-02T05:02:30.438ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
18072025-03-02T05:02:30.438ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 14 reason = Timeout
18082025-03-02T05:02:30.438ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18092025-03-02T05:02:30.438ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18102025-03-02T05:02:30.438ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18112025-03-02T05:02:30.438ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
18122025-03-02T05:02:30.439ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3479, tv_nsec: 669177356 } })
18132025-03-02T05:02:30.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18142025-03-02T05:02:30.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18152025-03-02T05:02:30.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18162025-03-02T05:02:30.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
18172025-03-02T05:02:30.439ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3479, tv_nsec: 669310158 } })
18182025-03-02T05:02:30.439ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.935908ms iteration = 14 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
18192025-03-02T05:02:30.439ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
18202025-03-02T05:02:30.439ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
18212025-03-02T05:02:30.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18222025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18232025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18242025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 14 reason = Timeout
18252025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18262025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
18272025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18282025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
18292025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 14 reason = Timeout
18302025-03-02T05:02:30.441ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
18312025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 14 reason = Timeout
18322025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18332025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
18342025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 14 reason = Timeout
18352025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
18362025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 14 reason = Timeout
18372025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
18382025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 14 reason = Timeout
18392025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
18402025-03-02T05:02:30.441ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 14 reason = Timeout
18412025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
18422025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
18432025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 40 reason = Timeout
18442025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18452025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 14 reason = Timeout
18462025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
18472025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 14 reason = Timeout
18482025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
18492025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
18502025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
18512025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18522025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18532025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
18542025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
18552025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
18562025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
18572025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
18582025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18592025-03-02T05:02:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
18602025-03-02T05:02:30.443ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 14 reason = Timeout
18612025-03-02T05:02:30.443ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
18622025-03-02T05:02:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
18632025-03-02T05:02:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
18642025-03-02T05:02:30.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18652025-03-02T05:02:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18662025-03-02T05:02:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
18672025-03-02T05:02:30.444ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18682025-03-02T05:02:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18692025-03-02T05:02:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
18702025-03-02T05:02:30.445ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
18712025-03-02T05:02:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
18722025-03-02T05:02:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
18732025-03-02T05:02:30.446ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18742025-03-02T05:02:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18752025-03-02T05:02:30.447ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18762025-03-02T05:02:30.448ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18772025-03-02T05:02:30.450ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
18782025-03-02T05:02:30.450ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18792025-03-02T05:02:30.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18802025-03-02T05:02:30.451ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
18812025-03-02T05:02:30.451ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18822025-03-02T05:02:30.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18832025-03-02T05:02:30.453ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18842025-03-02T05:02:30.453ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18852025-03-02T05:02:30.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18862025-03-02T05:02:30.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18872025-03-02T05:02:30.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18882025-03-02T05:02:30.456ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
18892025-03-02T05:02:30.456ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
18902025-03-02T05:02:30.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
18912025-03-02T05:02:30.457ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18922025-03-02T05:02:30.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18932025-03-02T05:02:30.460ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18942025-03-02T05:02:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18952025-03-02T05:02:30.463ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 21.733123ms iteration = 40 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
18962025-03-02T05:02:30.464ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3479, tv_nsec: 694039150 } })
18972025-03-02T05:02:30.464ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
18982025-03-02T05:02:30.475ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
18992025-03-02T05:02:30.476ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 34.078314ms iteration = 14 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
19002025-03-02T05:02:30.504ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
19012025-03-02T05:02:30.504ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
19022025-03-02T05:02:30.504ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
19032025-03-02T05:02:30.504ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
19042025-03-02T05:02:30.510ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19052025-03-02T05:02:30.511ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
19062025-03-02T05:02:30.512ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
19072025-03-02T05:02:30.512ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
19082025-03-02T05:02:30.512ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
19092025-03-02T05:02:30.513ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
19102025-03-02T05:02:30.513ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19112025-03-02T05:02:30.513ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19122025-03-02T05:02:30.517ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 75.27904ms iteration = 14 status = {"errors":[],"finish_invoked_ok":[]}
19132025-03-02T05:02:30.517ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 74.776283ms iteration = 14 status = {"errors":[],"requests_created_ok":[]}
19142025-03-02T05:02:30.518ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19152025-03-02T05:02:30.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19162025-03-02T05:02:30.520ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19172025-03-02T05:02:30.520ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
19182025-03-02T05:02:30.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
19192025-03-02T05:02:30.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19202025-03-02T05:02:30.522ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19212025-03-02T05:02:30.524ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 82.041137ms iteration = 14 status = {"errors":[],"garbage_collect_requested":[]}
19222025-03-02T05:02:30.525ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
19232025-03-02T05:02:30.525ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19242025-03-02T05:02:30.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19252025-03-02T05:02:30.528ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 89.763319ms iteration = 14 status = 0
19262025-03-02T05:02:30.533ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
19272025-03-02T05:02:30.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
19282025-03-02T05:02:30.535ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19292025-03-02T05:02:30.537ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19302025-03-02T05:02:30.538ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
19312025-03-02T05:02:30.538ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
19322025-03-02T05:02:30.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
19332025-03-02T05:02:30.539ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19342025-03-02T05:02:30.541ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
19352025-03-02T05:02:30.541ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
19362025-03-02T05:02:30.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
19372025-03-02T05:02:30.541ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
19382025-03-02T05:02:30.541ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19392025-03-02T05:02:30.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19402025-03-02T05:02:30.543ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
19412025-03-02T05:02:30.543ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19422025-03-02T05:02:30.543ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae7fee9e-ad20-426c-a5b4-03b0cf5a97db", "content-length": "26", "date": "Sun, 02 Mar 2025 05:02:30 GMT"} })
19432025-03-02T05:02:30.543ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19442025-03-02T05:02:30.544ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
19452025-03-02T05:02:30.559ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
19462025-03-02T05:02:30.559ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
19472025-03-02T05:02:30.559ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
19482025-03-02T05:02:30.560ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
19492025-03-02T05:02:30.563ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
19502025-03-02T05:02:30.563ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 124.683434ms iteration = 40 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
19512025-03-02T05:02:30.565ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 123.968634ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
19522025-03-02T05:02:30.571ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
19532025-03-02T05:02:30.571ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 129.136832ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
19542025-03-02T05:02:30.571ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
19552025-03-02T05:02:30.583ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
19562025-03-02T05:02:30.583ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 144.565542ms iteration = 14 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
19572025-03-02T05:02:30.593ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
19582025-03-02T05:02:30.593ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
19592025-03-02T05:02:30.593ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 151.856287ms iteration = 14 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
19602025-03-02T05:02:30.601ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 159.534977ms iteration = 14 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
19612025-03-02T05:02:30.603ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
19622025-03-02T05:02:30.603ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19632025-03-02T05:02:30.604ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4df89c3-ed3e-4160-85fb-0a38b09ee72f", "content-length": "26", "date": "Sun, 02 Mar 2025 05:02:30 GMT"} })
19642025-03-02T05:02:30.604ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19652025-03-02T05:02:30.607ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
19662025-03-02T05:02:30.618ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19672025-03-02T05:02:30.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19682025-03-02T05:02:30.639ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
19692025-03-02T05:02:30.639ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "17d4cb8b-f716-473e-8ddd-e5a0ef6c81bb", "content-length": "1", "date": "Sun, 02 Mar 2025 05:02:30 GMT"} })
19702025-03-02T05:02:30.640ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
19712025-03-02T05:02:30.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
19722025-03-02T05:02:30.642ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 206.006472ms iteration = 14 status = null
19732025-03-02T05:02:30.646ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
19742025-03-02T05:02:30.646ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19752025-03-02T05:02:30.646ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "12707155-0b0d-49ff-a65d-46b2c1206ec5", "content-length": "26", "date": "Sun, 02 Mar 2025 05:02:30 GMT"} })
19762025-03-02T05:02:30.646ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19772025-03-02T05:02:30.648ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
19782025-03-02T05:02:30.654ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 216.060754ms iteration = 15 status = {"physical_disks_added":0}
19792025-03-02T05:02:30.688ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19802025-03-02T05:02:30.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19812025-03-02T05:02:30.690ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19822025-03-02T05:02:30.691ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
19832025-03-02T05:02:30.692ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19842025-03-02T05:02:30.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19852025-03-02T05:02:30.693ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19862025-03-02T05:02:30.694ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
19872025-03-02T05:02:30.694ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19882025-03-02T05:02:30.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19892025-03-02T05:02:30.795ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
19902025-03-02T05:02:30.795ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
19912025-03-02T05:02:30.795ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7e52d11-e0fd-440d-9c30-6b8cefba2466", "content-length": "372", "date": "Sun, 02 Mar 2025 05:02:30 GMT"} })
19922025-03-02T05:02:30.795ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19932025-03-02T05:02:30.795ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19942025-03-02T05:02:30.811ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
19952025-03-02T05:02:30.811ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bc953d4-d278-46ba-bc9b-d45daca04e16", "content-length": "691", "date": "Sun, 02 Mar 2025 05:02:30 GMT"} })
19962025-03-02T05:02:30.811ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
19972025-03-02T05:02:30.811ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
19982025-03-02T05:02:30.811ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
19992025-03-02T05:02:30.813ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 371.264175ms iteration = 14 status = {}
20002025-03-02T05:02:30.815ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 379.042327ms iteration = 14 status = {}
20012025-03-02T05:02:30.838ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
20022025-03-02T05:02:30.838ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
20032025-03-02T05:02:30.857ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20042025-03-02T05:02:30.858ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20052025-03-02T05:02:30.860ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
20062025-03-02T05:02:30.860ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20072025-03-02T05:02:30.860ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20082025-03-02T05:02:30.871ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
20092025-03-02T05:02:30.871ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "20cb7a37-8b0a-436e-bc17-1a6a8a483963", "content-length": "134", "date": "Sun, 02 Mar 2025 05:02:30 GMT"} })
20102025-03-02T05:02:30.871ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
20112025-03-02T05:02:30.871ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
20122025-03-02T05:02:30.871ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
20132025-03-02T05:02:30.871ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
20142025-03-02T05:02:30.871ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
20152025-03-02T05:02:30.871ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
20162025-03-02T05:02:30.871ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
20172025-03-02T05:02:30.871ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
20182025-03-02T05:02:30.871ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
20192025-03-02T05:02:30.871ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
20202025-03-02T05:02:30.872ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
20212025-03-02T05:02:30.872ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20222025-03-02T05:02:30.872ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20232025-03-02T05:02:30.873ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
20242025-03-02T05:02:30.873ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "110e3d52-4f81-489a-aa57-75519c55890e", "content-length": "2", "date": "Sun, 02 Mar 2025 05:02:30 GMT"} })
20252025-03-02T05:02:30.873ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
20262025-03-02T05:02:30.874ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "75407536-77a9-4029-be17-54825212bc0b", "content-length": "52", "date": "Sun, 02 Mar 2025 05:02:30 GMT"} })
20272025-03-02T05:02:30.874ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
20282025-03-02T05:02:30.874ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
20292025-03-02T05:02:30.914ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "a7555850-14b2-4303-b868-575b2d201400", "date": "Sun, 02 Mar 2025 05:02:30 GMT"} })
20302025-03-02T05:02:30.914ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20312025-03-02T05:02:30.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20322025-03-02T05:02:30.916ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20332025-03-02T05:02:30.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20342025-03-02T05:02:30.919ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20352025-03-02T05:02:30.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20362025-03-02T05:02:30.922ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
20372025-03-02T05:02:30.922ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
20382025-03-02T05:02:30.926ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "91ce382e-f49c-4e6c-8ae6-2dfb31698fad", "date": "Sun, 02 Mar 2025 05:02:30 GMT"} })
20392025-03-02T05:02:30.928ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 486.913132ms iteration = 14 status = {}
20402025-03-02T05:02:31.008ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "1d821c00-653e-4da6-a4c7-c5beb4e7c1e8", "date": "Sun, 02 Mar 2025 05:02:30 GMT"} })
20412025-03-02T05:02:31.008ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
20422025-03-02T05:02:31.008ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
20432025-03-02T05:02:31.008ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "859d45d6-4679-473a-94f3-9eabecab8f6f", "content-length": "682", "date": "Sun, 02 Mar 2025 05:02:30 GMT"} })
20442025-03-02T05:02:31.008ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20452025-03-02T05:02:31.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20462025-03-02T05:02:31.012ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20472025-03-02T05:02:31.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20482025-03-02T05:02:31.016ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20492025-03-02T05:02:31.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20502025-03-02T05:02:31.019ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20512025-03-02T05:02:31.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20522025-03-02T05:02:31.026ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 584.776566ms iteration = 14 status = {}
20532025-03-02T05:02:32.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 280 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = a26c4bd0-c6b6-4ab8-978e-9cf85fed0fbf response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
20542025-03-02T05:02:40.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
20552025-03-02T05:02:40.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20562025-03-02T05:02:40.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20572025-03-02T05:02:40.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20582025-03-02T05:02:40.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 41 reason = Timeout
20592025-03-02T05:02:40.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20602025-03-02T05:02:40.443ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
20612025-03-02T05:02:40.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20622025-03-02T05:02:40.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
20632025-03-02T05:02:40.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20642025-03-02T05:02:40.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20652025-03-02T05:02:40.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
20662025-03-02T05:02:40.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20672025-03-02T05:02:40.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20682025-03-02T05:02:40.446ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.750458ms iteration = 41 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
20692025-03-02T05:02:40.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20702025-03-02T05:02:40.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
20712025-03-02T05:02:40.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20722025-03-02T05:02:40.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20732025-03-02T05:02:40.460ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
20742025-03-02T05:02:40.460ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.112875ms iteration = 41 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
20752025-03-02T05:02:42.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 290 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = eed01463-1233-47ee-a02b-7c7f863bb0a9 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
20762025-03-02T05:02:50.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
20772025-03-02T05:02:50.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20782025-03-02T05:02:50.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20792025-03-02T05:02:50.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20802025-03-02T05:02:50.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 42 reason = Timeout
20812025-03-02T05:02:50.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20822025-03-02T05:02:50.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
20832025-03-02T05:02:50.443ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
20842025-03-02T05:02:50.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20852025-03-02T05:02:50.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20862025-03-02T05:02:50.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20872025-03-02T05:02:50.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
20882025-03-02T05:02:50.445ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.392424ms iteration = 42 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
20892025-03-02T05:02:50.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20902025-03-02T05:02:50.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20912025-03-02T05:02:50.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20922025-03-02T05:02:50.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
20932025-03-02T05:02:50.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20942025-03-02T05:02:50.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20952025-03-02T05:02:50.458ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
20962025-03-02T05:02:50.458ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.423283ms iteration = 42 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
20972025-03-02T05:02:52.901ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 259 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = d0cf79b9-594a-4e21-914f-a51a1ddf05be response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
20982025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
20992025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 8 reason = Timeout
21002025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
21012025-03-02T05:03:00.436ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-02 04:53:00.436662516 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
21022025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
21032025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21042025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
21052025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
21062025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
21072025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21082025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
21092025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
21102025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
21112025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
21122025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
21132025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
21142025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
21152025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21162025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
21172025-03-02T05:03:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
21182025-03-02T05:03:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
21192025-03-02T05:03:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21202025-03-02T05:03:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21212025-03-02T05:03:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
21222025-03-02T05:03:00.437ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4a50e69a-1735-4503-96cb-55c6cc854815.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 3509, tv_nsec: 666983307 } })
21232025-03-02T05:03:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21242025-03-02T05:03:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
21252025-03-02T05:03:00.437ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3bf56dce-8353-46bb-a2f0-982fbd876e0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("742da26a-88eb-42ac-9329-e125fe787858.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 3509, tv_nsec: 667145920 } })
21262025-03-02T05:03:00.437ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
21272025-03-02T05:03:00.437ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 888.561\u{b5}s iteration = 8 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
21282025-03-02T05:03:00.437ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
21292025-03-02T05:03:00.437ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.044204ms iteration = 8 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
21302025-03-02T05:03:00.438ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 16 reason = Timeout
21312025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21322025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
21332025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
21342025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 8 reason = Timeout
21352025-03-02T05:03:00.439ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
21362025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 15 reason = Timeout
21372025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
21382025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21392025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
21402025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
21412025-03-02T05:03:00.439ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca
21422025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 9.51\u{b5}s iteration = 9 status = {"enabled":false,"target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca"}
21432025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
21442025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21452025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
21462025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21472025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
21482025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
21492025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21502025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21512025-03-02T05:03:00.439ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21522025-03-02T05:03:00.439ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3509, tv_nsec: 669362869 } })
21532025-03-02T05:03:00.439ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
21542025-03-02T05:03:00.440ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.921511ms iteration = 15 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
21552025-03-02T05:03:00.440ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21562025-03-02T05:03:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21572025-03-02T05:03:00.441ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
21582025-03-02T05:03:00.441ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21592025-03-02T05:03:00.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21602025-03-02T05:03:00.441ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3509, tv_nsec: 671478436 } })
21612025-03-02T05:03:00.441ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
21622025-03-02T05:03:00.441ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-02 04:53:00.436662516 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
21632025-03-02T05:03:00.441ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.257658ms iteration = 8 status = {"expiration":"2025-03-02T04:53:00.436662516Z","pruned":[]}
21642025-03-02T05:03:00.441ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 15 reason = Timeout
21652025-03-02T05:03:00.441ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
21662025-03-02T05:03:00.441ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 8 reason = Timeout
21672025-03-02T05:03:00.441ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
21682025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 15 reason = Timeout
21692025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
21702025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 43 reason = Timeout
21712025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21722025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 15 reason = Timeout
21732025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21742025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 15 reason = Timeout
21752025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
21762025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 15 reason = Timeout
21772025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
21782025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 15 reason = Timeout
21792025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
21802025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 15 reason = Timeout
21812025-03-02T05:03:00.442ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
21822025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 8 reason = Timeout
21832025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21842025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
21852025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 15 reason = Timeout
21862025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
21872025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 8 reason = Timeout
21882025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
21892025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 15 reason = Timeout
21902025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
21912025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
21922025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
21932025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
21942025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21952025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21962025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
21972025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
21982025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
21992025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
22002025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
22012025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
22022025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22032025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
22042025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22052025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
22062025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
22072025-03-02T05:03:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
22082025-03-02T05:03:00.443ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22092025-03-02T05:03:00.443ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22102025-03-02T05:03:00.443ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 15 reason = Timeout
22112025-03-02T05:03:00.443ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
22122025-03-02T05:03:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
22132025-03-02T05:03:00.443ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22142025-03-02T05:03:00.444ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22152025-03-02T05:03:00.444ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22162025-03-02T05:03:00.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
22172025-03-02T05:03:00.447ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
22182025-03-02T05:03:00.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
22192025-03-02T05:03:00.447ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22202025-03-02T05:03:00.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
22212025-03-02T05:03:00.447ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
22222025-03-02T05:03:00.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
22232025-03-02T05:03:00.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
22242025-03-02T05:03:00.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
22252025-03-02T05:03:00.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22262025-03-02T05:03:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
22272025-03-02T05:03:00.448ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22282025-03-02T05:03:00.449ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22292025-03-02T05:03:00.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22302025-03-02T05:03:00.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22312025-03-02T05:03:00.450ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22322025-03-02T05:03:00.450ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
22332025-03-02T05:03:00.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
22342025-03-02T05:03:00.451ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22352025-03-02T05:03:00.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22362025-03-02T05:03:00.452ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
22372025-03-02T05:03:00.452ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22382025-03-02T05:03:00.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22392025-03-02T05:03:00.453ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
22402025-03-02T05:03:00.453ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22412025-03-02T05:03:00.453ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
22422025-03-02T05:03:00.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22432025-03-02T05:03:00.455ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 13.122871ms iteration = 8 status = {"errors":[],"found_port_ok":[]}
22442025-03-02T05:03:00.455ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 13.369464ms iteration = 15 status = {"errors":[],"garbage_collect_requested":[]}
22452025-03-02T05:03:00.457ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22462025-03-02T05:03:00.457ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
22472025-03-02T05:03:00.457ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
22482025-03-02T05:03:00.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
22492025-03-02T05:03:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
22502025-03-02T05:03:00.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
22512025-03-02T05:03:00.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
22522025-03-02T05:03:00.460ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
22532025-03-02T05:03:00.460ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22542025-03-02T05:03:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22552025-03-02T05:03:00.467ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22562025-03-02T05:03:00.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22572025-03-02T05:03:00.473ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22582025-03-02T05:03:00.473ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22592025-03-02T05:03:00.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22602025-03-02T05:03:00.475ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
22612025-03-02T05:03:00.475ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
22622025-03-02T05:03:00.475ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3509, tv_nsec: 705528389 } })
22632025-03-02T05:03:00.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
22642025-03-02T05:03:00.476ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
22652025-03-02T05:03:00.476ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
22662025-03-02T05:03:00.476ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
22672025-03-02T05:03:00.476ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
22682025-03-02T05:03:00.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
22692025-03-02T05:03:00.484ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22702025-03-02T05:03:00.485ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 42.270031ms iteration = 15 status = {"errors":[],"requests_created_ok":[]}
22712025-03-02T05:03:00.485ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22722025-03-02T05:03:00.486ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
22732025-03-02T05:03:00.486ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22742025-03-02T05:03:00.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22752025-03-02T05:03:00.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22762025-03-02T05:03:00.489ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 47.486868ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
22772025-03-02T05:03:00.491ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22782025-03-02T05:03:00.495ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
22792025-03-02T05:03:00.495ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22802025-03-02T05:03:00.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22812025-03-02T05:03:00.514ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 72.684036ms iteration = 15 status = {"errors":[],"finish_invoked_ok":[]}
22822025-03-02T05:03:00.516ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
22832025-03-02T05:03:00.516ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
22842025-03-02T05:03:00.516ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
22852025-03-02T05:03:00.516ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
22862025-03-02T05:03:00.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
22872025-03-02T05:03:00.518ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22882025-03-02T05:03:00.520ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
22892025-03-02T05:03:00.520ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
22902025-03-02T05:03:00.520ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
22912025-03-02T05:03:00.520ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
22922025-03-02T05:03:00.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
22932025-03-02T05:03:00.523ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
22942025-03-02T05:03:00.524ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
22952025-03-02T05:03:00.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
22962025-03-02T05:03:00.525ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22972025-03-02T05:03:00.527ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
22982025-03-02T05:03:00.527ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 88.497023ms iteration = 43 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
22992025-03-02T05:03:00.528ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
23002025-03-02T05:03:00.528ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 0 version = 2
23012025-03-02T05:03:00.528ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
23022025-03-02T05:03:00.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
23032025-03-02T05:03:00.529ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
23042025-03-02T05:03:00.529ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
23052025-03-02T05:03:00.529ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 87.58078ms iteration = 8 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
23062025-03-02T05:03:00.530ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
23072025-03-02T05:03:00.530ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
23082025-03-02T05:03:00.530ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
23092025-03-02T05:03:00.531ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
23102025-03-02T05:03:00.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
23112025-03-02T05:03:00.534ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
23122025-03-02T05:03:00.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
23132025-03-02T05:03:00.538ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23142025-03-02T05:03:00.541ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 99.319713ms iteration = 43 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
23152025-03-02T05:03:00.541ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
23162025-03-02T05:03:00.541ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
23172025-03-02T05:03:00.541ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 102.777978ms iteration = 8 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
23182025-03-02T05:03:00.542ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
23192025-03-02T05:03:00.542ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
23202025-03-02T05:03:00.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
23212025-03-02T05:03:00.543ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23222025-03-02T05:03:00.544ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23232025-03-02T05:03:00.545ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
23242025-03-02T05:03:00.545ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
23252025-03-02T05:03:00.546ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23262025-03-02T05:03:00.549ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
23272025-03-02T05:03:00.549ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23282025-03-02T05:03:00.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23292025-03-02T05:03:00.554ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23302025-03-02T05:03:00.557ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
23312025-03-02T05:03:00.557ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23322025-03-02T05:03:00.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23332025-03-02T05:03:00.561ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
23342025-03-02T05:03:00.561ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
23352025-03-02T05:03:00.561ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 119.454235ms iteration = 15 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
23362025-03-02T05:03:00.565ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23372025-03-02T05:03:00.566ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23382025-03-02T05:03:00.569ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23392025-03-02T05:03:00.569ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23402025-03-02T05:03:00.575ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
23412025-03-02T05:03:00.575ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 133.328166ms iteration = 15 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
23422025-03-02T05:03:00.589ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
23432025-03-02T05:03:00.590ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
23442025-03-02T05:03:00.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23452025-03-02T05:03:00.593ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
23462025-03-02T05:03:00.593ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
23472025-03-02T05:03:00.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
23482025-03-02T05:03:00.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
23492025-03-02T05:03:00.595ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 152.962751ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
23502025-03-02T05:03:00.595ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23512025-03-02T05:03:00.596ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
23522025-03-02T05:03:00.596ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
23532025-03-02T05:03:00.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
23542025-03-02T05:03:00.600ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
23552025-03-02T05:03:00.600ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
23562025-03-02T05:03:00.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
23572025-03-02T05:03:00.602ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 160.5213ms iteration = 15 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
23582025-03-02T05:03:00.602ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 1 version = 2
23592025-03-02T05:03:00.602ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
23602025-03-02T05:03:00.602ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-02 04:56:26.100923 UTC dns_group = external generation = 2
23612025-03-02T05:03:00.602ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 166.156494ms iteration = 8 status = {"generation":2}
23622025-03-02T05:03:00.603ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
23632025-03-02T05:03:00.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
23642025-03-02T05:03:00.604ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
23652025-03-02T05:03:00.604ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
23662025-03-02T05:03:00.604ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
23672025-03-02T05:03:00.604ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-02 04:56:25.487890 UTC dns_group = internal generation = 1
23682025-03-02T05:03:00.604ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 167.975887ms iteration = 8 status = {"generation":1}
23692025-03-02T05:03:00.605ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 166.300105ms iteration = 16 status = {"physical_disks_added":0}
23702025-03-02T05:03:00.606ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23712025-03-02T05:03:00.608ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
23722025-03-02T05:03:00.608ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
23732025-03-02T05:03:00.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
23742025-03-02T05:03:00.610ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
23752025-03-02T05:03:00.610ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 171.324821ms iteration = 15 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
23762025-03-02T05:03:00.611ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
23772025-03-02T05:03:00.611ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
23782025-03-02T05:03:00.611ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ea4c6d2-2eb8-4f6f-9372-a45873ad0a6d", "content-length": "26", "date": "Sun, 02 Mar 2025 05:03:00 GMT"} })
23792025-03-02T05:03:00.611ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
23802025-03-02T05:03:00.613ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
23812025-03-02T05:03:00.621ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23822025-03-02T05:03:00.623ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
23832025-03-02T05:03:00.623ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23842025-03-02T05:03:00.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23852025-03-02T05:03:00.638ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23862025-03-02T05:03:00.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23872025-03-02T05:03:00.643ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
23882025-03-02T05:03:00.643ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
23892025-03-02T05:03:00.643ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a018bce-1c0b-47f8-8dc0-b9bc5a304e93", "content-length": "26", "date": "Sun, 02 Mar 2025 05:03:00 GMT"} })
23902025-03-02T05:03:00.643ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
23912025-03-02T05:03:00.645ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
23922025-03-02T05:03:00.709ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
23932025-03-02T05:03:00.709ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 267.017347ms iteration = 8 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
23942025-03-02T05:03:00.758ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
23952025-03-02T05:03:00.758ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
23962025-03-02T05:03:00.759ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e941245d-e1da-4e6f-b421-805128d78b20", "content-length": "26", "date": "Sun, 02 Mar 2025 05:03:00 GMT"} })
23972025-03-02T05:03:00.759ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
23982025-03-02T05:03:00.760ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
23992025-03-02T05:03:00.779ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
24002025-03-02T05:03:00.791ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
24012025-03-02T05:03:00.791ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
24022025-03-02T05:03:00.792ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ffa6442-4453-4d69-8e47-9aeee098346c", "content-length": "372", "date": "Sun, 02 Mar 2025 05:03:00 GMT"} })
24032025-03-02T05:03:00.792ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24042025-03-02T05:03:00.792ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24052025-03-02T05:03:00.794ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 355.48667ms iteration = 15 status = 0
24062025-03-02T05:03:00.799ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24072025-03-02T05:03:00.799ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24082025-03-02T05:03:00.832ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24092025-03-02T05:03:00.832ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-02T04:55:57.280988Z, time_modified: 2025-03-02T04:55:57.280988Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }) }
24102025-03-02T05:03:00.832ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 395.869946ms iteration = 8 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]}},"default_endpoint":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
24112025-03-02T05:03:00.834ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
24122025-03-02T05:03:00.834ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24132025-03-02T05:03:00.834ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
24142025-03-02T05:03:00.834ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
24152025-03-02T05:03:00.834ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24162025-03-02T05:03:00.834ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "ae6cac8b-457a-4d3c-ba87-a97fda97829c", "date": "Sun, 02 Mar 2025 05:03:00 GMT"} })
24172025-03-02T05:03:00.834ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24182025-03-02T05:03:00.834ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24192025-03-02T05:03:00.835ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24202025-03-02T05:03:00.836ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24212025-03-02T05:03:00.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24222025-03-02T05:03:00.838ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
24232025-03-02T05:03:00.838ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24242025-03-02T05:03:00.838ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24252025-03-02T05:03:00.839ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24262025-03-02T05:03:00.839ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24272025-03-02T05:03:00.842ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
24282025-03-02T05:03:00.842ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
24292025-03-02T05:03:00.843ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "0f7945af-b07d-4c13-b52e-b51cf58ff720", "date": "Sun, 02 Mar 2025 05:03:00 GMT"} })
24302025-03-02T05:03:00.844ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 402.39085ms iteration = 15 status = {}
24312025-03-02T05:03:00.936ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
24322025-03-02T05:03:00.941ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 505.090878ms iteration = 15 status = {}
24332025-03-02T05:03:00.954ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
24342025-03-02T05:03:00.955ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "92fa3674-e8d0-4732-863d-2f378c91acb5", "content-length": "1", "date": "Sun, 02 Mar 2025 05:02:59 GMT"} })
24352025-03-02T05:03:00.956ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
24362025-03-02T05:03:00.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
24372025-03-02T05:03:00.958ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 521.586453ms iteration = 15 status = null
24382025-03-02T05:03:01.067ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
24392025-03-02T05:03:01.067ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "fc4c3ed2-e12e-4ba2-80f9-43b81373c7bf", "date": "Sun, 02 Mar 2025 05:03:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
24402025-03-02T05:03:01.067ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
24412025-03-02T05:03:01.069ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "2ff9625d-730d-4eb5-b1cf-9769914ae063", "date": "Sun, 02 Mar 2025 05:03:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
24422025-03-02T05:03:01.069ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
24432025-03-02T05:03:01.071ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
24442025-03-02T05:03:01.071ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 634.463893ms iteration = 10 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
24452025-03-02T05:03:01.118ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
24462025-03-02T05:03:01.118ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "3600ba17-ef9c-4c8b-bdf4-0764870a70f3", "content-length": "691", "date": "Sun, 02 Mar 2025 05:03:00 GMT"} })
24472025-03-02T05:03:01.118ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
24482025-03-02T05:03:01.118ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
24492025-03-02T05:03:01.118ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
24502025-03-02T05:03:01.119ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 677.671497ms iteration = 15 status = {}
24512025-03-02T05:03:01.132ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
24522025-03-02T05:03:01.132ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "25e857a9-0061-4e20-8d53-46434d72ead4", "date": "Sun, 02 Mar 2025 05:03:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
24532025-03-02T05:03:01.132ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
24542025-03-02T05:03:01.134ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "2f20f824-5008-4ce1-ae68-91c0212c0575", "date": "Sun, 02 Mar 2025 05:03:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
24552025-03-02T05:03:01.134ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
24562025-03-02T05:03:01.135ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "7c8b6523-05ed-44b8-a995-0875cc134a98", "date": "Sun, 02 Mar 2025 05:03:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
24572025-03-02T05:03:01.135ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
24582025-03-02T05:03:01.137ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
24592025-03-02T05:03:01.137ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 700.666446ms iteration = 10 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
24602025-03-02T05:03:01.162ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24612025-03-02T05:03:01.162ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24622025-03-02T05:03:01.164ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
24632025-03-02T05:03:01.164ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24642025-03-02T05:03:01.164ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24652025-03-02T05:03:01.174ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
24662025-03-02T05:03:01.175ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec5c3077-7963-4f22-b4a7-eb108f1ce340", "content-length": "134", "date": "Sun, 02 Mar 2025 05:03:01 GMT"} })
24672025-03-02T05:03:01.175ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
24682025-03-02T05:03:01.175ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
24692025-03-02T05:03:01.175ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
24702025-03-02T05:03:01.175ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
24712025-03-02T05:03:01.175ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24722025-03-02T05:03:01.175ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24732025-03-02T05:03:01.175ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24742025-03-02T05:03:01.175ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
24752025-03-02T05:03:01.175ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24762025-03-02T05:03:01.175ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
24772025-03-02T05:03:01.175ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
24782025-03-02T05:03:01.175ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24792025-03-02T05:03:01.176ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24802025-03-02T05:03:01.177ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
24812025-03-02T05:03:01.178ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "55163320-17e6-4675-a420-05abff14f4c0", "content-length": "2", "date": "Sun, 02 Mar 2025 05:03:01 GMT"} })
24822025-03-02T05:03:01.178ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
24832025-03-02T05:03:01.178ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "23e81816-1be4-4066-a6ce-04ccc1798f69", "content-length": "52", "date": "Sun, 02 Mar 2025 05:03:01 GMT"} })
24842025-03-02T05:03:01.178ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
24852025-03-02T05:03:01.178ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
24862025-03-02T05:03:01.319ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "f74204d5-e6eb-40b8-bd01-978da0df8170", "date": "Sun, 02 Mar 2025 05:03:00 GMT"} })
24872025-03-02T05:03:01.319ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
24882025-03-02T05:03:01.319ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
24892025-03-02T05:03:01.319ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "6367e360-774e-4904-81cb-6b246e5b0126", "content-length": "682", "date": "Sun, 02 Mar 2025 05:03:01 GMT"} })
24902025-03-02T05:03:01.319ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24912025-03-02T05:03:01.319ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24922025-03-02T05:03:01.323ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24932025-03-02T05:03:01.323ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24942025-03-02T05:03:01.327ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24952025-03-02T05:03:01.327ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24962025-03-02T05:03:01.329ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24972025-03-02T05:03:01.329ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24982025-03-02T05:03:01.336ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 894.205337ms iteration = 15 status = {}
24992025-03-02T05:03:02.901ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 252 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = cd547dda-c188-4698-8fc9-95ba06c9ab1f response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
25002025-03-02T05:03:10.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
25012025-03-02T05:03:10.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25022025-03-02T05:03:10.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25032025-03-02T05:03:10.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25042025-03-02T05:03:10.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 44 reason = Timeout
25052025-03-02T05:03:10.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25062025-03-02T05:03:10.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25072025-03-02T05:03:10.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25082025-03-02T05:03:10.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25092025-03-02T05:03:10.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25102025-03-02T05:03:10.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25112025-03-02T05:03:10.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25122025-03-02T05:03:10.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25132025-03-02T05:03:10.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
25142025-03-02T05:03:10.450ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25152025-03-02T05:03:10.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25162025-03-02T05:03:10.463ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
25172025-03-02T05:03:10.463ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.198492ms iteration = 44 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
25182025-03-02T05:03:10.472ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25192025-03-02T05:03:10.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25202025-03-02T05:03:10.501ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 58.531773ms iteration = 44 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
25212025-03-02T05:03:12.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 291 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 1ce49a60-020b-4f6d-8976-6067f2a07ac9 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
25222025-03-02T05:03:20.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
25232025-03-02T05:03:20.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25242025-03-02T05:03:20.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25252025-03-02T05:03:20.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25262025-03-02T05:03:20.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 45 reason = Timeout
25272025-03-02T05:03:20.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25282025-03-02T05:03:20.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25292025-03-02T05:03:20.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25302025-03-02T05:03:20.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25312025-03-02T05:03:20.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25322025-03-02T05:03:20.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25332025-03-02T05:03:20.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25342025-03-02T05:03:20.472ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25352025-03-02T05:03:20.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25362025-03-02T05:03:20.474ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.51304ms iteration = 45 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
25372025-03-02T05:03:20.476ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25382025-03-02T05:03:20.478ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
25392025-03-02T05:03:20.478ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25402025-03-02T05:03:20.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25412025-03-02T05:03:20.490ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
25422025-03-02T05:03:20.490ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 50.382286ms iteration = 45 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
25432025-03-02T05:03:22.901ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 283 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 7e483011-d6d1-40b5-9f19-81da0a59e295 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
25442025-03-02T05:03:27.894ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:43671
25452025-03-02T05:03:27.894ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43671 req_id = ae61cfc0-f0bd-4490-8bda-1406b136fdac roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
25462025-03-02T05:03:27.895ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43671 req_id = ae61cfc0-f0bd-4490-8bda-1406b136fdac resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
25472025-03-02T05:03:27.897ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3356 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43671 req_id = ae61cfc0-f0bd-4490-8bda-1406b136fdac response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
25482025-03-02T05:03:27.898ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43671 req_id = 7d4032e0-3b0a-454a-bf3c-de1a7f995ba3 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
25492025-03-02T05:03:27.898ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43671 req_id = 7d4032e0-3b0a-454a-bf3c-de1a7f995ba3 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
25502025-03-02T05:03:27.900ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2083 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43671 req_id = 7d4032e0-3b0a-454a-bf3c-de1a7f995ba3 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
25512025-03-02T05:03:30.436ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
25522025-03-02T05:03:30.436ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
25532025-03-02T05:03:30.436ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
25542025-03-02T05:03:30.436ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
25552025-03-02T05:03:30.436ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
25562025-03-02T05:03:30.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
25572025-03-02T05:03:30.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
25582025-03-02T05:03:30.440ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 16 reason = Timeout
25592025-03-02T05:03:30.440ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
25602025-03-02T05:03:30.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
25612025-03-02T05:03:30.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25622025-03-02T05:03:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
25632025-03-02T05:03:30.440ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 17 reason = Timeout
25642025-03-02T05:03:30.440ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
25652025-03-02T05:03:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25662025-03-02T05:03:30.440ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
25672025-03-02T05:03:30.440ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25682025-03-02T05:03:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
25692025-03-02T05:03:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25702025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25712025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 16 reason = Timeout
25722025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
25732025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 16 reason = Timeout
25742025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 16 reason = Timeout
25752025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
25762025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25772025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
25782025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
25792025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
25802025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 16 reason = Timeout
25812025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 46 reason = Timeout
25822025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25832025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
25842025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 16 reason = Timeout
25852025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
25862025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 16 reason = Timeout
25872025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
25882025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 16 reason = Timeout
25892025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
25902025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 16 reason = Timeout
25912025-03-02T05:03:30.442ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
25922025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25932025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 16 reason = Timeout
25942025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
25952025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
25962025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25972025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
25982025-03-02T05:03:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
25992025-03-02T05:03:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26002025-03-02T05:03:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
26012025-03-02T05:03:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
26022025-03-02T05:03:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
26032025-03-02T05:03:30.443ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 6.675307ms iteration = 16 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
26042025-03-02T05:03:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26052025-03-02T05:03:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26062025-03-02T05:03:30.443ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3539, tv_nsec: 672726704 } })
26072025-03-02T05:03:30.443ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 16 reason = Timeout
26082025-03-02T05:03:30.443ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
26092025-03-02T05:03:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
26102025-03-02T05:03:30.443ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3539, tv_nsec: 672873176 } })
26112025-03-02T05:03:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
26122025-03-02T05:03:30.443ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
26132025-03-02T05:03:30.443ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
26142025-03-02T05:03:30.444ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26152025-03-02T05:03:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
26162025-03-02T05:03:30.444ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
26172025-03-02T05:03:30.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26182025-03-02T05:03:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
26192025-03-02T05:03:30.445ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26202025-03-02T05:03:30.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
26212025-03-02T05:03:30.447ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
26222025-03-02T05:03:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
26232025-03-02T05:03:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
26242025-03-02T05:03:30.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26252025-03-02T05:03:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26262025-03-02T05:03:30.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
26272025-03-02T05:03:30.449ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26282025-03-02T05:03:30.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26292025-03-02T05:03:30.450ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26302025-03-02T05:03:30.453ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
26312025-03-02T05:03:30.453ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26322025-03-02T05:03:30.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26332025-03-02T05:03:30.455ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26342025-03-02T05:03:30.458ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
26352025-03-02T05:03:30.458ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
26362025-03-02T05:03:30.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
26372025-03-02T05:03:30.462ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26382025-03-02T05:03:30.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26392025-03-02T05:03:30.463ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
26402025-03-02T05:03:30.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
26412025-03-02T05:03:30.468ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26422025-03-02T05:03:30.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26432025-03-02T05:03:30.469ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3539, tv_nsec: 699231999 } })
26442025-03-02T05:03:30.469ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
26452025-03-02T05:03:30.496ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 54.054084ms iteration = 16 status = {"errors":[],"finish_invoked_ok":[]}
26462025-03-02T05:03:30.503ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
26472025-03-02T05:03:30.504ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
26482025-03-02T05:03:30.506ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
26492025-03-02T05:03:30.506ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
26502025-03-02T05:03:30.508ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
26512025-03-02T05:03:30.508ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
26522025-03-02T05:03:30.508ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
26532025-03-02T05:03:30.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
26542025-03-02T05:03:30.511ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 69.09156ms iteration = 16 status = {"errors":[],"garbage_collect_requested":[]}
26552025-03-02T05:03:30.513ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
26562025-03-02T05:03:30.513ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
26572025-03-02T05:03:30.513ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26582025-03-02T05:03:30.514ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
26592025-03-02T05:03:30.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
26602025-03-02T05:03:30.516ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
26612025-03-02T05:03:30.516ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
26622025-03-02T05:03:30.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
26632025-03-02T05:03:30.516ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26642025-03-02T05:03:30.516ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26652025-03-02T05:03:30.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26662025-03-02T05:03:30.520ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
26672025-03-02T05:03:30.520ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26682025-03-02T05:03:30.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26692025-03-02T05:03:30.523ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26702025-03-02T05:03:30.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26712025-03-02T05:03:30.525ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
26722025-03-02T05:03:30.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
26732025-03-02T05:03:30.526ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26742025-03-02T05:03:30.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26752025-03-02T05:03:30.527ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
26762025-03-02T05:03:30.527ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
26772025-03-02T05:03:30.527ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26782025-03-02T05:03:30.528ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a33cfea-d9df-415b-b2e3-44ce83fa1d70", "content-length": "26", "date": "Sun, 02 Mar 2025 05:03:30 GMT"} })
26792025-03-02T05:03:30.528ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
26802025-03-02T05:03:30.529ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
26812025-03-02T05:03:30.530ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
26822025-03-02T05:03:30.530ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
26832025-03-02T05:03:30.530ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
26842025-03-02T05:03:30.532ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
26852025-03-02T05:03:30.532ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 90.327035ms iteration = 16 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
26862025-03-02T05:03:30.535ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 92.620615ms iteration = 16 status = {"errors":[],"requests_created_ok":[]}
26872025-03-02T05:03:30.540ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26882025-03-02T05:03:30.540ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
26892025-03-02T05:03:30.540ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
26902025-03-02T05:03:30.541ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0fc0e78-08d5-4809-9c92-f213fc3387ac", "content-length": "26", "date": "Sun, 02 Mar 2025 05:03:30 GMT"} })
26912025-03-02T05:03:30.541ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
26922025-03-02T05:03:30.543ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
26932025-03-02T05:03:30.544ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
26942025-03-02T05:03:30.544ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26952025-03-02T05:03:30.544ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26962025-03-02T05:03:30.552ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 109.764909ms iteration = 16 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
26972025-03-02T05:03:30.552ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
26982025-03-02T05:03:30.552ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
26992025-03-02T05:03:30.552ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c12ca7b-c115-4ad4-9f6a-ef06bcf4b2d2", "content-length": "26", "date": "Sun, 02 Mar 2025 05:03:30 GMT"} })
27002025-03-02T05:03:30.552ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
27012025-03-02T05:03:30.554ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
27022025-03-02T05:03:30.555ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
27032025-03-02T05:03:30.555ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
27042025-03-02T05:03:30.555ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
27052025-03-02T05:03:30.556ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
27062025-03-02T05:03:30.557ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
27072025-03-02T05:03:30.557ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 117.491509ms iteration = 46 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
27082025-03-02T05:03:30.558ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 115.798408ms iteration = 46 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
27092025-03-02T05:03:30.559ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 116.810251ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
27102025-03-02T05:03:30.564ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 121.902017ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
27112025-03-02T05:03:30.565ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27122025-03-02T05:03:30.566ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27132025-03-02T05:03:30.567ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
27142025-03-02T05:03:30.567ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27152025-03-02T05:03:30.567ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27162025-03-02T05:03:30.569ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
27172025-03-02T05:03:30.569ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27182025-03-02T05:03:30.569ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27192025-03-02T05:03:30.571ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 131.209219ms iteration = 16 status = 0
27202025-03-02T05:03:30.572ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
27212025-03-02T05:03:30.572ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 132.637137ms iteration = 16 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
27222025-03-02T05:03:30.588ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
27232025-03-02T05:03:30.588ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
27242025-03-02T05:03:30.588ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 146.374766ms iteration = 16 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
27252025-03-02T05:03:30.601ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
27262025-03-02T05:03:30.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
27272025-03-02T05:03:30.611ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
27282025-03-02T05:03:30.611ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7ca3170-02e1-49ba-ab6b-bd5f126e849f", "content-length": "1", "date": "Sun, 02 Mar 2025 05:03:30 GMT"} })
27292025-03-02T05:03:30.614ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
27302025-03-02T05:03:30.614ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
27312025-03-02T05:03:30.618ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 181.652086ms iteration = 16 status = null
27322025-03-02T05:03:30.638ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 202.084511ms iteration = 16 status = {}
27332025-03-02T05:03:30.639ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 199.639529ms iteration = 17 status = {"physical_disks_added":0}
27342025-03-02T05:03:30.706ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
27352025-03-02T05:03:30.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
27362025-03-02T05:03:30.716ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
27372025-03-02T05:03:30.716ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
27382025-03-02T05:03:30.725ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "77dbbf32-87a2-42aa-8431-af494b485ee2", "content-length": "372", "date": "Sun, 02 Mar 2025 05:03:30 GMT"} })
27392025-03-02T05:03:30.725ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27402025-03-02T05:03:30.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27412025-03-02T05:03:30.730ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27422025-03-02T05:03:30.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27432025-03-02T05:03:30.732ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
27442025-03-02T05:03:30.732ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27452025-03-02T05:03:30.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27462025-03-02T05:03:30.740ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
27472025-03-02T05:03:30.740ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
27482025-03-02T05:03:30.741ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
27492025-03-02T05:03:30.742ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "81c0a20f-72e9-4bbc-a3ea-379780bc0c8f", "content-length": "134", "date": "Sun, 02 Mar 2025 05:03:30 GMT"} })
27502025-03-02T05:03:30.742ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
27512025-03-02T05:03:30.742ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
27522025-03-02T05:03:30.742ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
27532025-03-02T05:03:30.742ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
27542025-03-02T05:03:30.742ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
27552025-03-02T05:03:30.742ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
27562025-03-02T05:03:30.742ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
27572025-03-02T05:03:30.742ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
27582025-03-02T05:03:30.742ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
27592025-03-02T05:03:30.742ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
27602025-03-02T05:03:30.742ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
27612025-03-02T05:03:30.742ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27622025-03-02T05:03:30.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27632025-03-02T05:03:30.744ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
27642025-03-02T05:03:30.745ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa98287f-20e1-4f1e-97c2-e29f20888b58", "content-length": "2", "date": "Sun, 02 Mar 2025 05:03:30 GMT"} })
27652025-03-02T05:03:30.745ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
27662025-03-02T05:03:30.745ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "e859d3cd-5b6b-4f75-8caf-474f44b8b68e", "content-length": "52", "date": "Sun, 02 Mar 2025 05:03:30 GMT"} })
27672025-03-02T05:03:30.745ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
27682025-03-02T05:03:30.745ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
27692025-03-02T05:03:30.746ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27702025-03-02T05:03:30.747ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
27712025-03-02T05:03:30.747ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
27722025-03-02T05:03:30.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
27732025-03-02T05:03:30.749ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27742025-03-02T05:03:30.751ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
27752025-03-02T05:03:30.751ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
27762025-03-02T05:03:30.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
27772025-03-02T05:03:30.785ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
27782025-03-02T05:03:30.824ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "42073272-9da1-4eb8-acd0-b19a45bcea42", "date": "Sun, 02 Mar 2025 05:03:30 GMT"} })
27792025-03-02T05:03:30.824ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27802025-03-02T05:03:30.824ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27812025-03-02T05:03:30.826ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27822025-03-02T05:03:30.826ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27832025-03-02T05:03:30.828ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e14480e-b765-4f2a-ba9b-b939c1b45e61", "content-length": "691", "date": "Sun, 02 Mar 2025 05:03:30 GMT"} })
27842025-03-02T05:03:30.828ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
27852025-03-02T05:03:30.828ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
27862025-03-02T05:03:30.828ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
27872025-03-02T05:03:30.828ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 386.26786ms iteration = 16 status = {}
27882025-03-02T05:03:30.829ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27892025-03-02T05:03:30.830ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27902025-03-02T05:03:30.833ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
27912025-03-02T05:03:30.833ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
27922025-03-02T05:03:30.833ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "48d93786-b24b-4642-9267-1b635287b5c6", "date": "Sun, 02 Mar 2025 05:03:30 GMT"} })
27932025-03-02T05:03:30.834ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 392.426831ms iteration = 16 status = {}
27942025-03-02T05:03:30.914ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "11db67a6-cd68-4774-820c-c15e5b768c8c", "date": "Sun, 02 Mar 2025 05:03:30 GMT"} })
27952025-03-02T05:03:30.914ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
27962025-03-02T05:03:30.914ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
27972025-03-02T05:03:30.915ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1a9a24b-76b7-4236-8a01-fd2deb0fed8c", "content-length": "682", "date": "Sun, 02 Mar 2025 05:03:30 GMT"} })
27982025-03-02T05:03:30.915ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27992025-03-02T05:03:30.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28002025-03-02T05:03:30.918ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28012025-03-02T05:03:30.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28022025-03-02T05:03:30.921ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28032025-03-02T05:03:30.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28042025-03-02T05:03:30.923ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28052025-03-02T05:03:30.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28062025-03-02T05:03:30.931ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 488.835557ms iteration = 16 status = {}
28072025-03-02T05:03:32.901ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 289 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = c4243788-4b54-486b-81e2-0f051774e1b8 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
28082025-03-02T05:03:40.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
28092025-03-02T05:03:40.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28102025-03-02T05:03:40.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28112025-03-02T05:03:40.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28122025-03-02T05:03:40.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 47 reason = Timeout
28132025-03-02T05:03:40.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
28142025-03-02T05:03:40.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
28152025-03-02T05:03:40.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
28162025-03-02T05:03:40.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28172025-03-02T05:03:40.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28182025-03-02T05:03:40.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28192025-03-02T05:03:40.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28202025-03-02T05:03:40.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28212025-03-02T05:03:40.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
28222025-03-02T05:03:40.450ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28232025-03-02T05:03:40.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28242025-03-02T05:03:40.461ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
28252025-03-02T05:03:40.461ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.755673ms iteration = 47 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
28262025-03-02T05:03:40.472ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
28272025-03-02T05:03:40.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
28282025-03-02T05:03:40.502ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.631097ms iteration = 47 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
28292025-03-02T05:03:42.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 265 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = fc17dcd3-8227-402a-a558-0fa5d110abeb response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
28302025-03-02T05:03:50.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
28312025-03-02T05:03:50.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28322025-03-02T05:03:50.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28332025-03-02T05:03:50.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28342025-03-02T05:03:50.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 48 reason = Timeout
28352025-03-02T05:03:50.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
28362025-03-02T05:03:50.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
28372025-03-02T05:03:50.443ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
28382025-03-02T05:03:50.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28392025-03-02T05:03:50.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28402025-03-02T05:03:50.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
28412025-03-02T05:03:50.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
28422025-03-02T05:03:50.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28432025-03-02T05:03:50.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28442025-03-02T05:03:50.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28452025-03-02T05:03:50.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
28462025-03-02T05:03:50.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28472025-03-02T05:03:50.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28482025-03-02T05:03:50.460ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
28492025-03-02T05:03:50.460ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.661573ms iteration = 48 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
28502025-03-02T05:03:50.474ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.740933ms iteration = 48 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
28512025-03-02T05:03:52.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 425 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 1b323e5a-73f5-4ac6-9107-0e7561d4d28e response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
28522025-03-02T05:03:57.895ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43671 req_id = 5bf86eca-b111-455f-b91b-c9c50dc33ae0 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
28532025-03-02T05:03:57.895ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43671 req_id = 5bf86eca-b111-455f-b91b-c9c50dc33ae0 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
28542025-03-02T05:03:57.898ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3314 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43671 req_id = 5bf86eca-b111-455f-b91b-c9c50dc33ae0 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
28552025-03-02T05:03:57.899ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43671 req_id = c5ec96ae-42db-42a5-bb4c-d33361e24710 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
28562025-03-02T05:03:57.899ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43671 req_id = c5ec96ae-42db-42a5-bb4c-d33361e24710 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
28572025-03-02T05:03:57.901ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2491 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43671 req_id = c5ec96ae-42db-42a5-bb4c-d33361e24710 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
28582025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
28592025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 9 reason = Timeout
28602025-03-02T05:04:00.437ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-02 04:54:00.437192742 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
28612025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
28622025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
28632025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
28642025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
28652025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
28662025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
28672025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
28682025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
28692025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
28702025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
28712025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
28722025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
28732025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
28742025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
28752025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
28762025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
28772025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
28782025-03-02T05:04:00.437ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3bf56dce-8353-46bb-a2f0-982fbd876e0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("742da26a-88eb-42ac-9329-e125fe787858.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 3569, tv_nsec: 667130212 } })
28792025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
28802025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
28812025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28822025-03-02T05:04:00.437ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4a50e69a-1735-4503-96cb-55c6cc854815.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 3569, tv_nsec: 667235364 } })
28832025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28842025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
28852025-03-02T05:04:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
28862025-03-02T05:04:00.438ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
28872025-03-02T05:04:00.438ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
28882025-03-02T05:04:00.438ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.016693ms iteration = 9 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
28892025-03-02T05:04:00.438ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
28902025-03-02T05:04:00.438ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.082724ms iteration = 9 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
28912025-03-02T05:04:00.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
28922025-03-02T05:04:00.439ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
28932025-03-02T05:04:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28942025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 18 reason = Timeout
28952025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
28962025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 9 reason = Timeout
28972025-03-02T05:04:00.440ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
28982025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 17 reason = Timeout
28992025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
29002025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
29012025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29022025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
29032025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
29042025-03-02T05:04:00.440ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca
29052025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 11.5\u{b5}s iteration = 10 status = {"enabled":false,"target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca"}
29062025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
29072025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29082025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
29092025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
29102025-03-02T05:04:00.440ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3569, tv_nsec: 669800107 } })
29112025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
29122025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29132025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29142025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29152025-03-02T05:04:00.440ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
29162025-03-02T05:04:00.440ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.755509ms iteration = 17 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
29172025-03-02T05:04:00.441ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-02 04:54:00.437192742 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
29182025-03-02T05:04:00.441ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.387117ms iteration = 9 status = {"expiration":"2025-03-02T04:54:00.437192742Z","pruned":[]}
29192025-03-02T05:04:00.441ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29202025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
29212025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
29222025-03-02T05:04:00.442ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3569, tv_nsec: 671627291 } })
29232025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
29242025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 17 reason = Timeout
29252025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 17 reason = Timeout
29262025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29272025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29282025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 17 reason = Timeout
29292025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 17 reason = Timeout
29302025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
29312025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
29322025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 9 reason = Timeout
29332025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
29342025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 17 reason = Timeout
29352025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
29362025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
29372025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 49 reason = Timeout
29382025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29392025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 9 reason = Timeout
29402025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
29412025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 9 reason = Timeout
29422025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
29432025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 17 reason = Timeout
29442025-03-02T05:04:00.442ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
29452025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29462025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 17 reason = Timeout
29472025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 17 reason = Timeout
29482025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
29492025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
29502025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 17 reason = Timeout
29512025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
29522025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
29532025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
29542025-03-02T05:04:00.442ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
29552025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29562025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29572025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
29582025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
29592025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
29602025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29612025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
29622025-03-02T05:04:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
29632025-03-02T05:04:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
29642025-03-02T05:04:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
29652025-03-02T05:04:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
29662025-03-02T05:04:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
29672025-03-02T05:04:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
29682025-03-02T05:04:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
29692025-03-02T05:04:00.443ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 17 reason = Timeout
29702025-03-02T05:04:00.443ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
29712025-03-02T05:04:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
29722025-03-02T05:04:00.444ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29732025-03-02T05:04:00.444ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29742025-03-02T05:04:00.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
29752025-03-02T05:04:00.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
29762025-03-02T05:04:00.446ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
29772025-03-02T05:04:00.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
29782025-03-02T05:04:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
29792025-03-02T05:04:00.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29802025-03-02T05:04:00.450ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29812025-03-02T05:04:00.451ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
29822025-03-02T05:04:00.451ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29832025-03-02T05:04:00.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
29842025-03-02T05:04:00.454ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
29852025-03-02T05:04:00.454ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29862025-03-02T05:04:00.454ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
29872025-03-02T05:04:00.454ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29882025-03-02T05:04:00.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29892025-03-02T05:04:00.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29902025-03-02T05:04:00.455ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
29912025-03-02T05:04:00.455ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
29922025-03-02T05:04:00.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
29932025-03-02T05:04:00.457ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 14.701471ms iteration = 9 status = {"errors":[],"found_port_ok":[]}
29942025-03-02T05:04:00.489ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 45.310991ms iteration = 17 status = {"errors":[],"requests_created_ok":[]}
29952025-03-02T05:04:00.494ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3569, tv_nsec: 723523177 } })
29962025-03-02T05:04:00.494ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
29972025-03-02T05:04:00.494ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29982025-03-02T05:04:00.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29992025-03-02T05:04:00.511ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
30002025-03-02T05:04:00.511ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
30012025-03-02T05:04:00.511ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
30022025-03-02T05:04:00.511ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
30032025-03-02T05:04:00.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
30042025-03-02T05:04:00.512ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30052025-03-02T05:04:00.512ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30062025-03-02T05:04:00.513ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30072025-03-02T05:04:00.513ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30082025-03-02T05:04:00.516ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
30092025-03-02T05:04:00.516ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
30102025-03-02T05:04:00.516ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
30112025-03-02T05:04:00.516ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
30122025-03-02T05:04:00.516ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
30132025-03-02T05:04:00.516ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
30142025-03-02T05:04:00.517ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
30152025-03-02T05:04:00.517ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
30162025-03-02T05:04:00.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
30172025-03-02T05:04:00.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
30182025-03-02T05:04:00.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
30192025-03-02T05:04:00.518ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
30202025-03-02T05:04:00.518ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30212025-03-02T05:04:00.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30222025-03-02T05:04:00.518ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
30232025-03-02T05:04:00.518ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
30242025-03-02T05:04:00.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
30252025-03-02T05:04:00.522ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30262025-03-02T05:04:00.522ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30272025-03-02T05:04:00.524ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
30282025-03-02T05:04:00.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
30292025-03-02T05:04:00.527ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
30302025-03-02T05:04:00.527ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
30312025-03-02T05:04:00.527ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
30322025-03-02T05:04:00.527ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
30332025-03-02T05:04:00.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
30342025-03-02T05:04:00.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
30352025-03-02T05:04:00.528ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
30362025-03-02T05:04:00.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
30372025-03-02T05:04:00.530ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
30382025-03-02T05:04:00.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
30392025-03-02T05:04:00.531ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 88.988328ms iteration = 49 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
30402025-03-02T05:04:00.533ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
30412025-03-02T05:04:00.533ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 0 version = 2
30422025-03-02T05:04:00.533ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
30432025-03-02T05:04:00.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
30442025-03-02T05:04:00.533ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30452025-03-02T05:04:00.533ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
30462025-03-02T05:04:00.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
30472025-03-02T05:04:00.536ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
30482025-03-02T05:04:00.537ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
30492025-03-02T05:04:00.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
30502025-03-02T05:04:00.543ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 100.745482ms iteration = 17 status = {"errors":[],"garbage_collect_requested":[]}
30512025-03-02T05:04:00.544ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
30522025-03-02T05:04:00.544ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
30532025-03-02T05:04:00.544ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 102.257142ms iteration = 9 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
30542025-03-02T05:04:00.545ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
30552025-03-02T05:04:00.545ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
30562025-03-02T05:04:00.545ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
30572025-03-02T05:04:00.545ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
30582025-03-02T05:04:00.545ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30592025-03-02T05:04:00.553ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30602025-03-02T05:04:00.554ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
30612025-03-02T05:04:00.554ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
30622025-03-02T05:04:00.554ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
30632025-03-02T05:04:00.554ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
30642025-03-02T05:04:00.555ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30652025-03-02T05:04:00.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
30662025-03-02T05:04:00.555ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
30672025-03-02T05:04:00.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
30682025-03-02T05:04:00.556ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 113.747861ms iteration = 17 status = {"errors":[],"finish_invoked_ok":[]}
30692025-03-02T05:04:00.557ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
30702025-03-02T05:04:00.557ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 1 version = 2
30712025-03-02T05:04:00.557ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
30722025-03-02T05:04:00.557ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-02 04:56:26.100923 UTC dns_group = external generation = 2
30732025-03-02T05:04:00.557ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 120.198785ms iteration = 9 status = {"generation":2}
30742025-03-02T05:04:00.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
30752025-03-02T05:04:00.557ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
30762025-03-02T05:04:00.557ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
30772025-03-02T05:04:00.558ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
30782025-03-02T05:04:00.558ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
30792025-03-02T05:04:00.558ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
30802025-03-02T05:04:00.558ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
30812025-03-02T05:04:00.558ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
30822025-03-02T05:04:00.559ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30832025-03-02T05:04:00.561ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
30842025-03-02T05:04:00.561ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
30852025-03-02T05:04:00.561ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
30862025-03-02T05:04:00.562ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
30872025-03-02T05:04:00.562ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30882025-03-02T05:04:00.562ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30892025-03-02T05:04:00.567ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 124.575232ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
30902025-03-02T05:04:00.568ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30912025-03-02T05:04:00.569ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
30922025-03-02T05:04:00.570ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
30932025-03-02T05:04:00.570ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 127.944506ms iteration = 17 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
30942025-03-02T05:04:00.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
30952025-03-02T05:04:00.571ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
30962025-03-02T05:04:00.571ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30972025-03-02T05:04:00.571ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30982025-03-02T05:04:00.572ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30992025-03-02T05:04:00.572ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
31002025-03-02T05:04:00.572ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 132.453826ms iteration = 49 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
31012025-03-02T05:04:00.574ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
31022025-03-02T05:04:00.574ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
31032025-03-02T05:04:00.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
31042025-03-02T05:04:00.587ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31052025-03-02T05:04:00.587ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
31062025-03-02T05:04:00.587ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
31072025-03-02T05:04:00.587ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 147.431761ms iteration = 9 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
31082025-03-02T05:04:00.587ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31092025-03-02T05:04:00.589ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31102025-03-02T05:04:00.590ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31112025-03-02T05:04:00.592ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31122025-03-02T05:04:00.595ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
31132025-03-02T05:04:00.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
31142025-03-02T05:04:00.596ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
31152025-03-02T05:04:00.596ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31162025-03-02T05:04:00.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31172025-03-02T05:04:00.597ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
31182025-03-02T05:04:00.597ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
31192025-03-02T05:04:00.597ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
31202025-03-02T05:04:00.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
31212025-03-02T05:04:00.612ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
31222025-03-02T05:04:00.612ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
31232025-03-02T05:04:00.613ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6033e706-21f1-4eba-bf91-fc2b6c87d9be", "content-length": "26", "date": "Sun, 02 Mar 2025 05:04:00 GMT"} })
31242025-03-02T05:04:00.613ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
31252025-03-02T05:04:00.614ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 171.598695ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
31262025-03-02T05:04:00.614ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
31272025-03-02T05:04:00.614ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 174.747136ms iteration = 17 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
31282025-03-02T05:04:00.614ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
31292025-03-02T05:04:00.614ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
31302025-03-02T05:04:00.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
31312025-03-02T05:04:00.626ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
31322025-03-02T05:04:00.626ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
31332025-03-02T05:04:00.626ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 183.607052ms iteration = 17 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
31342025-03-02T05:04:00.629ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
31352025-03-02T05:04:00.629ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
31362025-03-02T05:04:00.629ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bde6e342-f712-4c12-b1a4-5eebe6f47b6d", "content-length": "26", "date": "Sun, 02 Mar 2025 05:04:00 GMT"} })
31372025-03-02T05:04:00.629ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
31382025-03-02T05:04:00.632ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
31392025-03-02T05:04:00.645ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-02T04:55:57.280988Z, time_modified: 2025-03-02T04:55:57.280988Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }) }
31402025-03-02T05:04:00.645ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 208.445635ms iteration = 9 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]}},"default_endpoint":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
31412025-03-02T05:04:00.648ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31422025-03-02T05:04:00.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
31432025-03-02T05:04:00.697ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 257.755498ms iteration = 18 status = {"physical_disks_added":0}
31442025-03-02T05:04:00.741ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
31452025-03-02T05:04:00.741ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 299.128036ms iteration = 9 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
31462025-03-02T05:04:00.764ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
31472025-03-02T05:04:00.764ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
31482025-03-02T05:04:00.764ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed7826ee-2ab8-4f3e-bcce-feda3b11f40a", "content-length": "372", "date": "Sun, 02 Mar 2025 05:04:00 GMT"} })
31492025-03-02T05:04:00.764ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31502025-03-02T05:04:00.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31512025-03-02T05:04:00.766ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31522025-03-02T05:04:00.766ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 326.508143ms iteration = 17 status = 0
31532025-03-02T05:04:00.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
31542025-03-02T05:04:00.769ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
31552025-03-02T05:04:00.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
31562025-03-02T05:04:00.772ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
31572025-03-02T05:04:00.772ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
31582025-03-02T05:04:00.772ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-02 04:56:25.487890 UTC dns_group = internal generation = 1
31592025-03-02T05:04:00.772ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 335.566681ms iteration = 9 status = {"generation":1}
31602025-03-02T05:04:00.786ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
31612025-03-02T05:04:00.786ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "965ad974-78e2-4f1b-9a60-7dbd7f51b4d7", "content-length": "1", "date": "Sun, 02 Mar 2025 05:03:59 GMT"} })
31622025-03-02T05:04:00.788ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
31632025-03-02T05:04:00.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
31642025-03-02T05:04:00.789ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
31652025-03-02T05:04:00.789ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
31662025-03-02T05:04:00.789ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "6ba7b86a-86b8-4c2a-a42a-c0970f7f8a20", "date": "Sun, 02 Mar 2025 05:04:00 GMT"} })
31672025-03-02T05:04:00.789ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31682025-03-02T05:04:00.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31692025-03-02T05:04:00.790ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 353.432013ms iteration = 17 status = null
31702025-03-02T05:04:00.792ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31712025-03-02T05:04:00.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31722025-03-02T05:04:00.796ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31732025-03-02T05:04:00.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31742025-03-02T05:04:00.799ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
31752025-03-02T05:04:00.799ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
31762025-03-02T05:04:00.800ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "85e44a74-d4de-4bc6-bf51-920d020c7860", "date": "Sun, 02 Mar 2025 05:04:00 GMT"} })
31772025-03-02T05:04:00.802ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 359.960129ms iteration = 17 status = {}
31782025-03-02T05:04:00.812ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 369.732256ms iteration = 17 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
31792025-03-02T05:04:00.817ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 380.589287ms iteration = 17 status = {}
31802025-03-02T05:04:00.821ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
31812025-03-02T05:04:00.822ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
31822025-03-02T05:04:00.880ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
31832025-03-02T05:04:00.880ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
31842025-03-02T05:04:00.880ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "aeb75411-7fd5-44f7-bc2b-fbb0869930c2", "content-length": "26", "date": "Sun, 02 Mar 2025 05:04:00 GMT"} })
31852025-03-02T05:04:00.880ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
31862025-03-02T05:04:00.882ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
31872025-03-02T05:04:00.888ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31882025-03-02T05:04:00.888ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
31892025-03-02T05:04:00.889ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31902025-03-02T05:04:00.892ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
31912025-03-02T05:04:00.892ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31922025-03-02T05:04:00.892ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
31932025-03-02T05:04:00.893ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31942025-03-02T05:04:00.896ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
31952025-03-02T05:04:00.896ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31962025-03-02T05:04:00.896ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
31972025-03-02T05:04:01.031ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
31982025-03-02T05:04:01.031ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "3730d690-a977-4d9d-b215-d264c44051a1", "date": "Sun, 02 Mar 2025 05:04:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
31992025-03-02T05:04:01.031ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
32002025-03-02T05:04:01.032ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "fd1b8016-fa7e-4595-a424-2602fa1269d7", "date": "Sun, 02 Mar 2025 05:04:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
32012025-03-02T05:04:01.032ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
32022025-03-02T05:04:01.034ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
32032025-03-02T05:04:01.034ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 597.470662ms iteration = 11 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
32042025-03-02T05:04:01.042ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
32052025-03-02T05:04:01.046ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
32062025-03-02T05:04:01.090ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3579592-1ad3-4569-9e13-33f15fbbc5ea", "content-length": "691", "date": "Sun, 02 Mar 2025 05:04:00 GMT"} })
32072025-03-02T05:04:01.090ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
32082025-03-02T05:04:01.090ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
32092025-03-02T05:04:01.090ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
32102025-03-02T05:04:01.092ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 649.825575ms iteration = 17 status = {}
32112025-03-02T05:04:01.115ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
32122025-03-02T05:04:01.115ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "005c410f-a9ad-4019-89b5-b039fffc2f6d", "date": "Sun, 02 Mar 2025 05:04:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
32132025-03-02T05:04:01.115ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
32142025-03-02T05:04:01.117ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "9e02c79a-2f7e-458a-ac8f-7d0259baac03", "date": "Sun, 02 Mar 2025 05:04:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
32152025-03-02T05:04:01.117ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
32162025-03-02T05:04:01.119ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "3e7de886-85b0-4200-b07c-6ac2c39bfbc8", "date": "Sun, 02 Mar 2025 05:04:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
32172025-03-02T05:04:01.119ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
32182025-03-02T05:04:01.121ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
32192025-03-02T05:04:01.121ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 684.309244ms iteration = 11 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
32202025-03-02T05:04:01.134ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32212025-03-02T05:04:01.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32222025-03-02T05:04:01.135ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
32232025-03-02T05:04:01.135ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32242025-03-02T05:04:01.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32252025-03-02T05:04:01.144ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
32262025-03-02T05:04:01.144ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3a5f5aa-8f67-4d6e-af56-cc9d880bf3df", "content-length": "134", "date": "Sun, 02 Mar 2025 05:04:00 GMT"} })
32272025-03-02T05:04:01.144ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
32282025-03-02T05:04:01.144ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
32292025-03-02T05:04:01.145ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
32302025-03-02T05:04:01.145ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
32312025-03-02T05:04:01.145ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
32322025-03-02T05:04:01.145ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
32332025-03-02T05:04:01.145ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
32342025-03-02T05:04:01.145ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
32352025-03-02T05:04:01.145ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
32362025-03-02T05:04:01.145ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
32372025-03-02T05:04:01.145ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
32382025-03-02T05:04:01.145ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32392025-03-02T05:04:01.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32402025-03-02T05:04:01.148ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
32412025-03-02T05:04:01.148ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "656d2b7b-df28-4cfb-8d72-62be507ee9c9", "content-length": "2", "date": "Sun, 02 Mar 2025 05:04:00 GMT"} })
32422025-03-02T05:04:01.148ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
32432025-03-02T05:04:01.148ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "89ec1b57-cc2f-4291-bfed-11ac7e74a758", "content-length": "52", "date": "Sun, 02 Mar 2025 05:04:00 GMT"} })
32442025-03-02T05:04:01.148ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
32452025-03-02T05:04:01.148ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
32462025-03-02T05:04:01.239ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "ae7b0bb7-59e3-4a63-89d2-d4da3b9277cf", "date": "Sun, 02 Mar 2025 05:04:01 GMT"} })
32472025-03-02T05:04:01.239ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
32482025-03-02T05:04:01.239ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
32492025-03-02T05:04:01.239ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbe6cf7e-5fc3-4c5f-8a32-75e81bc542bc", "content-length": "682", "date": "Sun, 02 Mar 2025 05:04:01 GMT"} })
32502025-03-02T05:04:01.239ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32512025-03-02T05:04:01.240ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32522025-03-02T05:04:01.243ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32532025-03-02T05:04:01.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32542025-03-02T05:04:01.247ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32552025-03-02T05:04:01.247ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32562025-03-02T05:04:01.249ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32572025-03-02T05:04:01.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32582025-03-02T05:04:01.257ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 814.711553ms iteration = 17 status = {}
32592025-03-02T05:04:02.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 273 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 07d41c6a-2b71-4db1-90bc-5b1026e77dd1 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
32602025-03-02T05:04:10.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
32612025-03-02T05:04:10.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32622025-03-02T05:04:10.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32632025-03-02T05:04:10.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32642025-03-02T05:04:10.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 50 reason = Timeout
32652025-03-02T05:04:10.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32662025-03-02T05:04:10.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32672025-03-02T05:04:10.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
32682025-03-02T05:04:10.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32692025-03-02T05:04:10.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32702025-03-02T05:04:10.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32712025-03-02T05:04:10.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32722025-03-02T05:04:10.446ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.677808ms iteration = 50 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
32732025-03-02T05:04:10.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32742025-03-02T05:04:10.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32752025-03-02T05:04:10.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32762025-03-02T05:04:10.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
32772025-03-02T05:04:10.450ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32782025-03-02T05:04:10.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32792025-03-02T05:04:10.461ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
32802025-03-02T05:04:10.461ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.819324ms iteration = 50 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
32812025-03-02T05:04:12.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 319 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 641f0a0e-3df2-4873-93f9-e9886a6c8283 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
32822025-03-02T05:04:20.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
32832025-03-02T05:04:20.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32842025-03-02T05:04:20.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32852025-03-02T05:04:20.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32862025-03-02T05:04:20.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 51 reason = Timeout
32872025-03-02T05:04:20.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32882025-03-02T05:04:20.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32892025-03-02T05:04:20.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
32902025-03-02T05:04:20.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32912025-03-02T05:04:20.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32922025-03-02T05:04:20.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32932025-03-02T05:04:20.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32942025-03-02T05:04:20.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32952025-03-02T05:04:20.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32962025-03-02T05:04:20.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32972025-03-02T05:04:20.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
32982025-03-02T05:04:20.450ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32992025-03-02T05:04:20.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33002025-03-02T05:04:20.463ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
33012025-03-02T05:04:20.463ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.333354ms iteration = 51 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
33022025-03-02T05:04:20.480ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 37.016452ms iteration = 51 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
33032025-03-02T05:04:22.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 263 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 44d71067-5f3c-4148-92b7-2d103e1850d0 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
33042025-03-02T05:04:30.436ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
33052025-03-02T05:04:30.436ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
33062025-03-02T05:04:30.436ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
33072025-03-02T05:04:30.436ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
33082025-03-02T05:04:30.436ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
33092025-03-02T05:04:30.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
33102025-03-02T05:04:30.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
33112025-03-02T05:04:30.438ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3599, tv_nsec: 668220499 } })
33122025-03-02T05:04:30.439ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
33132025-03-02T05:04:30.439ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 19 reason = Timeout
33142025-03-02T05:04:30.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 52 reason = Timeout
33152025-03-02T05:04:30.439ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 18 reason = Timeout
33162025-03-02T05:04:30.439ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
33172025-03-02T05:04:30.439ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
33182025-03-02T05:04:30.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33192025-03-02T05:04:30.439ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
33202025-03-02T05:04:30.439ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
33212025-03-02T05:04:30.439ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3599, tv_nsec: 669058960 } })
33222025-03-02T05:04:30.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33232025-03-02T05:04:30.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
33242025-03-02T05:04:30.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
33252025-03-02T05:04:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
33262025-03-02T05:04:30.440ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
33272025-03-02T05:04:30.440ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.652908ms iteration = 18 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
33282025-03-02T05:04:30.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33292025-03-02T05:04:30.441ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33302025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33312025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 18 reason = Timeout
33322025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33332025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 18 reason = Timeout
33342025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
33352025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33362025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 18 reason = Timeout
33372025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
33382025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
33392025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
33402025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 18 reason = Timeout
33412025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
33422025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 18 reason = Timeout
33432025-03-02T05:04:30.442ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
33442025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 18 reason = Timeout
33452025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33462025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
33472025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 18 reason = Timeout
33482025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
33492025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 18 reason = Timeout
33502025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
33512025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
33522025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 18 reason = Timeout
33532025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
33542025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 52 reason = Timeout
33552025-03-02T05:04:30.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33562025-03-02T05:04:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
33572025-03-02T05:04:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
33582025-03-02T05:04:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33592025-03-02T05:04:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
33602025-03-02T05:04:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
33612025-03-02T05:04:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
33622025-03-02T05:04:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
33632025-03-02T05:04:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
33642025-03-02T05:04:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33652025-03-02T05:04:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33662025-03-02T05:04:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
33672025-03-02T05:04:30.443ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 18 reason = Timeout
33682025-03-02T05:04:30.443ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
33692025-03-02T05:04:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
33702025-03-02T05:04:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
33712025-03-02T05:04:30.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33722025-03-02T05:04:30.444ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 2.24424ms iteration = 18 status = {"errors":[],"garbage_collect_requested":[]}
33732025-03-02T05:04:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33742025-03-02T05:04:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
33752025-03-02T05:04:30.445ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
33762025-03-02T05:04:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
33772025-03-02T05:04:30.445ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
33782025-03-02T05:04:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
33792025-03-02T05:04:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
33802025-03-02T05:04:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
33812025-03-02T05:04:30.446ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
33822025-03-02T05:04:30.446ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33832025-03-02T05:04:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
33842025-03-02T05:04:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
33852025-03-02T05:04:30.448ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33862025-03-02T05:04:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33872025-03-02T05:04:30.450ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33882025-03-02T05:04:30.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33892025-03-02T05:04:30.451ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33902025-03-02T05:04:30.452ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33912025-03-02T05:04:30.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33922025-03-02T05:04:30.453ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
33932025-03-02T05:04:30.453ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33942025-03-02T05:04:30.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33952025-03-02T05:04:30.454ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
33962025-03-02T05:04:30.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
33972025-03-02T05:04:30.457ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
33982025-03-02T05:04:30.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
33992025-03-02T05:04:30.460ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 18.259868ms iteration = 52 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
34002025-03-02T05:04:30.463ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34012025-03-02T05:04:30.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
34022025-03-02T05:04:30.465ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
34032025-03-02T05:04:30.465ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3599, tv_nsec: 695056278 } })
34042025-03-02T05:04:30.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
34052025-03-02T05:04:30.466ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
34062025-03-02T05:04:30.466ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34072025-03-02T05:04:30.466ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
34082025-03-02T05:04:30.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34092025-03-02T05:04:30.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34102025-03-02T05:04:30.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
34112025-03-02T05:04:30.470ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
34122025-03-02T05:04:30.470ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34132025-03-02T05:04:30.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34142025-03-02T05:04:30.473ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 29.94545ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
34152025-03-02T05:04:30.490ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
34162025-03-02T05:04:30.490ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 50.580709ms iteration = 52 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
34172025-03-02T05:04:30.497ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 54.989517ms iteration = 18 status = {"errors":[],"finish_invoked_ok":[]}
34182025-03-02T05:04:30.502ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
34192025-03-02T05:04:30.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
34202025-03-02T05:04:30.505ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34212025-03-02T05:04:30.506ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
34222025-03-02T05:04:30.507ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34232025-03-02T05:04:30.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
34242025-03-02T05:04:30.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34252025-03-02T05:04:30.508ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
34262025-03-02T05:04:30.508ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
34272025-03-02T05:04:30.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
34282025-03-02T05:04:30.509ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
34292025-03-02T05:04:30.509ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
34302025-03-02T05:04:30.509ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
34312025-03-02T05:04:30.509ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 66.05322ms iteration = 18 status = {"errors":[],"requests_created_ok":[]}
34322025-03-02T05:04:30.510ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34332025-03-02T05:04:30.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
34342025-03-02T05:04:30.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34352025-03-02T05:04:30.512ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34362025-03-02T05:04:30.513ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
34372025-03-02T05:04:30.513ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
34382025-03-02T05:04:30.515ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
34392025-03-02T05:04:30.515ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34402025-03-02T05:04:30.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34412025-03-02T05:04:30.516ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34422025-03-02T05:04:30.519ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 79.888071ms iteration = 18 status = 0
34432025-03-02T05:04:30.519ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
34442025-03-02T05:04:30.519ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
34452025-03-02T05:04:30.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
34462025-03-02T05:04:30.523ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34472025-03-02T05:04:30.526ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
34482025-03-02T05:04:30.526ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34492025-03-02T05:04:30.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34502025-03-02T05:04:30.548ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
34512025-03-02T05:04:30.548ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 108.806147ms iteration = 18 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
34522025-03-02T05:04:30.555ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
34532025-03-02T05:04:30.555ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
34542025-03-02T05:04:30.555ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
34552025-03-02T05:04:30.555ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
34562025-03-02T05:04:30.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
34572025-03-02T05:04:30.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
34582025-03-02T05:04:30.563ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 120.418238ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
34592025-03-02T05:04:30.566ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
34602025-03-02T05:04:30.566ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 123.702681ms iteration = 18 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
34612025-03-02T05:04:30.592ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
34622025-03-02T05:04:30.592ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
34632025-03-02T05:04:30.592ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 149.656879ms iteration = 18 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
34642025-03-02T05:04:30.595ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 152.921701ms iteration = 18 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
34652025-03-02T05:04:30.600ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34662025-03-02T05:04:30.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34672025-03-02T05:04:30.602ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34682025-03-02T05:04:30.604ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
34692025-03-02T05:04:30.604ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34702025-03-02T05:04:30.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34712025-03-02T05:04:30.604ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 165.346673ms iteration = 19 status = {"physical_disks_added":0}
34722025-03-02T05:04:30.728ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
34732025-03-02T05:04:30.728ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
34742025-03-02T05:04:30.728ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "20a27685-6bb7-419d-ab73-20551c018ab5", "content-length": "372", "date": "Sun, 02 Mar 2025 05:04:30 GMT"} })
34752025-03-02T05:04:30.728ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34762025-03-02T05:04:30.729ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34772025-03-02T05:04:30.730ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34782025-03-02T05:04:30.731ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
34792025-03-02T05:04:30.731ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34802025-03-02T05:04:30.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34812025-03-02T05:04:30.739ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
34822025-03-02T05:04:30.739ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
34832025-03-02T05:04:30.740ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "975ba3bc-8487-4f83-86a3-87f77511b564", "content-length": "26", "date": "Sun, 02 Mar 2025 05:04:30 GMT"} })
34842025-03-02T05:04:30.740ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
34852025-03-02T05:04:30.741ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
34862025-03-02T05:04:30.761ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
34872025-03-02T05:04:30.761ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
34882025-03-02T05:04:30.761ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9286050d-1858-4cc4-baf0-9d6606c51c5b", "content-length": "26", "date": "Sun, 02 Mar 2025 05:04:30 GMT"} })
34892025-03-02T05:04:30.761ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
34902025-03-02T05:04:30.762ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
34912025-03-02T05:04:30.762ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
34922025-03-02T05:04:30.762ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c82ac90-6861-42b6-b9ec-d17fea4cd0ce", "content-length": "26", "date": "Sun, 02 Mar 2025 05:04:30 GMT"} })
34932025-03-02T05:04:30.762ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
34942025-03-02T05:04:30.764ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
34952025-03-02T05:04:30.764ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
34962025-03-02T05:04:30.772ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
34972025-03-02T05:04:30.772ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
34982025-03-02T05:04:30.859ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
34992025-03-02T05:04:30.859ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "fbe320ea-e299-4ba0-ac48-067bef7a24cc", "date": "Sun, 02 Mar 2025 05:04:30 GMT"} })
35002025-03-02T05:04:30.859ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35012025-03-02T05:04:30.859ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "004a5b85-7958-44e1-b634-2e8b93a32f89", "content-length": "1", "date": "Sun, 02 Mar 2025 05:04:30 GMT"} })
35022025-03-02T05:04:30.859ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35032025-03-02T05:04:30.861ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
35042025-03-02T05:04:30.861ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
35052025-03-02T05:04:30.862ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35062025-03-02T05:04:30.862ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35072025-03-02T05:04:30.863ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 426.446185ms iteration = 18 status = null
35082025-03-02T05:04:30.866ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35092025-03-02T05:04:30.866ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35102025-03-02T05:04:30.869ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
35112025-03-02T05:04:30.869ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
35122025-03-02T05:04:30.869ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "fc90c982-3d06-40fa-b23a-f6f4b64cf869", "date": "Sun, 02 Mar 2025 05:04:30 GMT"} })
35132025-03-02T05:04:30.871ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 428.40966ms iteration = 18 status = {}
35142025-03-02T05:04:30.945ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
35152025-03-02T05:04:30.945ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "364ebaef-1679-4566-ad92-64b3521aa23a", "content-length": "691", "date": "Sun, 02 Mar 2025 05:04:30 GMT"} })
35162025-03-02T05:04:30.945ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
35172025-03-02T05:04:30.945ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
35182025-03-02T05:04:30.945ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
35192025-03-02T05:04:30.947ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 505.354313ms iteration = 18 status = {}
35202025-03-02T05:04:30.957ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 521.062507ms iteration = 18 status = {}
35212025-03-02T05:04:30.994ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35222025-03-02T05:04:30.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35232025-03-02T05:04:30.997ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
35242025-03-02T05:04:30.997ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35252025-03-02T05:04:30.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35262025-03-02T05:04:31.007ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
35272025-03-02T05:04:31.007ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd4efa38-b726-4254-a62b-0fce81149b91", "content-length": "134", "date": "Sun, 02 Mar 2025 05:04:30 GMT"} })
35282025-03-02T05:04:31.007ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
35292025-03-02T05:04:31.007ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
35302025-03-02T05:04:31.008ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
35312025-03-02T05:04:31.008ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
35322025-03-02T05:04:31.008ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
35332025-03-02T05:04:31.008ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
35342025-03-02T05:04:31.008ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
35352025-03-02T05:04:31.008ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
35362025-03-02T05:04:31.008ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
35372025-03-02T05:04:31.008ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
35382025-03-02T05:04:31.008ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
35392025-03-02T05:04:31.008ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35402025-03-02T05:04:31.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35412025-03-02T05:04:31.011ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
35422025-03-02T05:04:31.011ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6e47cd1-9b98-4c81-9c86-9c192da7cec2", "content-length": "2", "date": "Sun, 02 Mar 2025 05:04:30 GMT"} })
35432025-03-02T05:04:31.011ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
35442025-03-02T05:04:31.011ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "53e3314c-c4b7-41f3-b374-d946c32f2f1f", "content-length": "52", "date": "Sun, 02 Mar 2025 05:04:30 GMT"} })
35452025-03-02T05:04:31.011ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
35462025-03-02T05:04:31.011ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
35472025-03-02T05:04:31.101ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "f855d0d6-1785-45c7-a1e8-80cc15a986ac", "date": "Sun, 02 Mar 2025 05:04:30 GMT"} })
35482025-03-02T05:04:31.101ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
35492025-03-02T05:04:31.101ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
35502025-03-02T05:04:31.101ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "d204c730-5a0c-4037-a43a-bf6a297080d1", "content-length": "682", "date": "Sun, 02 Mar 2025 05:04:30 GMT"} })
35512025-03-02T05:04:31.101ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35522025-03-02T05:04:31.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35532025-03-02T05:04:31.106ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35542025-03-02T05:04:31.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35552025-03-02T05:04:31.109ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35562025-03-02T05:04:31.110ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35572025-03-02T05:04:31.112ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35582025-03-02T05:04:31.112ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35592025-03-02T05:04:31.119ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 677.307972ms iteration = 18 status = {}
35602025-03-02T05:04:31.182ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
35612025-03-02T05:04:31.182ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
35622025-03-02T05:04:31.212ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
35632025-03-02T05:04:31.212ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:58166
35642025-03-02T05:04:31.212ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:58166 req_id = 43b6dad8-cd0b-401a-b39d-815c66f1a2e1 roles = RoleSet { roles: {} } uri = /metrics/producers
35652025-03-02T05:04:31.213ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:58166 req_id = 43b6dad8-cd0b-401a-b39d-815c66f1a2e1 resource = Database result = Ok(()) uri = /metrics/producers
35662025-03-02T05:04:31.254ZINFOnexus: assigned collector to new producer collector_id = 8aaa536c-c534-4dd2-aad0-92d8934e5eba file = nexus/src/app/oximeter.rs:76 producer_id = aebed526-5e03-4a58-86b5-1baf17c88392
35672025-03-02T05:04:31.254ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 41926 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:58166 req_id = 43b6dad8-cd0b-401a-b39d-815c66f1a2e1 response_code = 201 uri = /metrics/producers
35682025-03-02T05:04:31.254ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "43b6dad8-cd0b-401a-b39d-815c66f1a2e1", "content-length": "41", "date": "Sun, 02 Mar 2025 05:04:31 GMT"} })
35692025-03-02T05:04:31.256ZDEBGnexus (producer-registration-task): registered with nexus successfully
35702025-03-02T05:04:31.256ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
35712025-03-02T05:04:32.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 259 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 1bec89ec-a563-4b71-8d42-8de4be7f8225 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
35722025-03-02T05:04:40.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 53 reason = Timeout
35732025-03-02T05:04:40.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35742025-03-02T05:04:40.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35752025-03-02T05:04:40.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35762025-03-02T05:04:40.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 53 reason = Timeout
35772025-03-02T05:04:40.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
35782025-03-02T05:04:40.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
35792025-03-02T05:04:40.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
35802025-03-02T05:04:40.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35812025-03-02T05:04:40.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35822025-03-02T05:04:40.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
35832025-03-02T05:04:40.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
35842025-03-02T05:04:40.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35852025-03-02T05:04:40.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35862025-03-02T05:04:40.446ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.593136ms iteration = 53 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
35872025-03-02T05:04:40.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35882025-03-02T05:04:40.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
35892025-03-02T05:04:40.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35902025-03-02T05:04:40.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35912025-03-02T05:04:40.456ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
35922025-03-02T05:04:40.456ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.145743ms iteration = 53 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
35932025-03-02T05:04:42.895ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:34481
35942025-03-02T05:04:42.895ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:34481 req_id = 7560aead-116b-4399-ac60-376622108e83 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
35952025-03-02T05:04:42.896ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:34481 req_id = 7560aead-116b-4399-ac60-376622108e83 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
35962025-03-02T05:04:42.898ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2921 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:34481 req_id = 7560aead-116b-4399-ac60-376622108e83 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
35972025-03-02T05:04:42.898ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:34481 req_id = 320a457a-7c93-4afc-bdff-25a99825db9c roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
35982025-03-02T05:04:42.899ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:34481 req_id = 320a457a-7c93-4afc-bdff-25a99825db9c resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
35992025-03-02T05:04:42.901ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2117 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:34481 req_id = 320a457a-7c93-4afc-bdff-25a99825db9c response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
36002025-03-02T05:04:42.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 223 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = c3a892f2-ede3-4e7c-b8b6-776f02191a5a response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
36012025-03-02T05:04:50.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 54 reason = Timeout
36022025-03-02T05:04:50.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36032025-03-02T05:04:50.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36042025-03-02T05:04:50.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36052025-03-02T05:04:50.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 54 reason = Timeout
36062025-03-02T05:04:50.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36072025-03-02T05:04:50.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36082025-03-02T05:04:50.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
36092025-03-02T05:04:50.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36102025-03-02T05:04:50.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36112025-03-02T05:04:50.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36122025-03-02T05:04:50.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36132025-03-02T05:04:50.446ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.738909ms iteration = 54 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
36142025-03-02T05:04:50.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36152025-03-02T05:04:50.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36162025-03-02T05:04:50.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36172025-03-02T05:04:50.451ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
36182025-03-02T05:04:50.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36192025-03-02T05:04:50.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36202025-03-02T05:04:50.460ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
36212025-03-02T05:04:50.460ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.635329ms iteration = 54 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
36222025-03-02T05:04:52.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 264 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = d4bdc53b-c2ab-48f0-a088-4067e8b6521c response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
36232025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 19 reason = Timeout
36242025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 10 reason = Timeout
36252025-03-02T05:05:00.436ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-02 04:55:00.436711793 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
36262025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 19 reason = Timeout
36272025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
36282025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 10 reason = Timeout
36292025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
36302025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
36312025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
36322025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 10 reason = Timeout
36332025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
36342025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 12 reason = Timeout
36352025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 10 reason = Timeout
36362025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
36372025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 12 reason = Timeout
36382025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
36392025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 10 reason = Timeout
36402025-03-02T05:05:00.436ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 19 reason = Timeout
36412025-03-02T05:05:00.437ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
36422025-03-02T05:05:00.437ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 10 reason = Timeout
36432025-03-02T05:05:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
36442025-03-02T05:05:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
36452025-03-02T05:05:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
36462025-03-02T05:05:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
36472025-03-02T05:05:00.437ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4a50e69a-1735-4503-96cb-55c6cc854815.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 3629, tv_nsec: 666413735 } })
36482025-03-02T05:05:00.437ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3bf56dce-8353-46bb-a2f0-982fbd876e0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("742da26a-88eb-42ac-9329-e125fe787858.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 3629, tv_nsec: 666446196 } })
36492025-03-02T05:05:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
36502025-03-02T05:05:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
36512025-03-02T05:05:00.437ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
36522025-03-02T05:05:00.437ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 937.602\u{b5}s iteration = 10 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
36532025-03-02T05:05:00.437ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
36542025-03-02T05:05:00.437ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.025354ms iteration = 10 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
36552025-03-02T05:05:00.439ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36562025-03-02T05:05:00.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 11 reason = Timeout
36572025-03-02T05:05:00.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 55 reason = Timeout
36582025-03-02T05:05:00.439ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 20 reason = Timeout
36592025-03-02T05:05:00.439ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca
36602025-03-02T05:05:00.439ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 14.12\u{b5}s iteration = 11 status = {"enabled":false,"target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca"}
36612025-03-02T05:05:00.439ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
36622025-03-02T05:05:00.439ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 5 reason = Timeout
36632025-03-02T05:05:00.439ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 10 reason = Timeout
36642025-03-02T05:05:00.439ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
36652025-03-02T05:05:00.439ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 19 reason = Timeout
36662025-03-02T05:05:00.439ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
36672025-03-02T05:05:00.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36682025-03-02T05:05:00.439ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36692025-03-02T05:05:00.439ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
36702025-03-02T05:05:00.439ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 19 reason = Timeout
36712025-03-02T05:05:00.439ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36722025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
36732025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
36742025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36752025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
36762025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36772025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
36782025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36792025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
36802025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
36812025-03-02T05:05:00.440ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3629, tv_nsec: 669419424 } })
36822025-03-02T05:05:00.440ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-02 04:55:00.436711793 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
36832025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.607657ms iteration = 10 status = {"expiration":"2025-03-02T04:55:00.436711793Z","pruned":[]}
36842025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
36852025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36862025-03-02T05:05:00.440ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
36872025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
36882025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36892025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.988812ms iteration = 19 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
36902025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
36912025-03-02T05:05:00.440ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36922025-03-02T05:05:00.441ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
36932025-03-02T05:05:00.441ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
36942025-03-02T05:05:00.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
36952025-03-02T05:05:00.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
36962025-03-02T05:05:00.441ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3629, tv_nsec: 670758242 } })
36972025-03-02T05:05:00.441ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
36982025-03-02T05:05:00.441ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
36992025-03-02T05:05:00.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
37002025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37012025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 19 reason = Timeout
37022025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
37032025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 10 reason = Timeout
37042025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
37052025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 19 reason = Timeout
37062025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
37072025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 19 reason = Timeout
37082025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
37092025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 19 reason = Timeout
37102025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
37112025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 19 reason = Timeout
37122025-03-02T05:05:00.442ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
37132025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 19 reason = Timeout
37142025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37152025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 19 reason = Timeout
37162025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
37172025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 10 reason = Timeout
37182025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
37192025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
37202025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 10 reason = Timeout
37212025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
37222025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 19 reason = Timeout
37232025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
37242025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 55 reason = Timeout
37252025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
37262025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 19 reason = Timeout
37272025-03-02T05:05:00.442ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
37282025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
37292025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 19 reason = Timeout
37302025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37312025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
37322025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
37332025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
37342025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
37352025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
37362025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
37372025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37382025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
37392025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
37402025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
37412025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
37422025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
37432025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37442025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
37452025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37462025-03-02T05:05:00.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37472025-03-02T05:05:00.444ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37482025-03-02T05:05:00.444ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 19 reason = Timeout
37492025-03-02T05:05:00.444ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
37502025-03-02T05:05:00.444ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37512025-03-02T05:05:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
37522025-03-02T05:05:00.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
37532025-03-02T05:05:00.445ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
37542025-03-02T05:05:00.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
37552025-03-02T05:05:00.445ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
37562025-03-02T05:05:00.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
37572025-03-02T05:05:00.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
37582025-03-02T05:05:00.447ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37592025-03-02T05:05:00.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
37602025-03-02T05:05:00.447ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37612025-03-02T05:05:00.447ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37622025-03-02T05:05:00.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37632025-03-02T05:05:00.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
37642025-03-02T05:05:00.447ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
37652025-03-02T05:05:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
37662025-03-02T05:05:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
37672025-03-02T05:05:00.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37682025-03-02T05:05:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37692025-03-02T05:05:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
37702025-03-02T05:05:00.448ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
37712025-03-02T05:05:00.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
37722025-03-02T05:05:00.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
37732025-03-02T05:05:00.450ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37742025-03-02T05:05:00.451ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
37752025-03-02T05:05:00.451ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37762025-03-02T05:05:00.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37772025-03-02T05:05:00.451ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37782025-03-02T05:05:00.451ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37792025-03-02T05:05:00.452ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
37802025-03-02T05:05:00.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
37812025-03-02T05:05:00.453ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 10.819801ms iteration = 19 status = {"errors":[],"finish_invoked_ok":[]}
37822025-03-02T05:05:00.453ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
37832025-03-02T05:05:00.453ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37842025-03-02T05:05:00.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37852025-03-02T05:05:00.453ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37862025-03-02T05:05:00.454ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
37872025-03-02T05:05:00.455ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
37882025-03-02T05:05:00.455ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37892025-03-02T05:05:00.455ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37902025-03-02T05:05:00.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37912025-03-02T05:05:00.456ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
37922025-03-02T05:05:00.456ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
37932025-03-02T05:05:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
37942025-03-02T05:05:00.457ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
37952025-03-02T05:05:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
37962025-03-02T05:05:00.457ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 15.149487ms iteration = 10 status = {"errors":[],"found_port_ok":[]}
37972025-03-02T05:05:00.457ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 15.134407ms iteration = 19 status = {"errors":[],"garbage_collect_requested":[]}
37982025-03-02T05:05:00.458ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37992025-03-02T05:05:00.459ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
38002025-03-02T05:05:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
38012025-03-02T05:05:00.460ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
38022025-03-02T05:05:00.460ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
38032025-03-02T05:05:00.460ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 17.491018ms iteration = 10 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
38042025-03-02T05:05:00.460ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
38052025-03-02T05:05:00.460ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38062025-03-02T05:05:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38072025-03-02T05:05:00.460ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
38082025-03-02T05:05:00.460ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
38092025-03-02T05:05:00.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
38102025-03-02T05:05:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
38112025-03-02T05:05:00.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
38122025-03-02T05:05:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
38132025-03-02T05:05:00.462ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38142025-03-02T05:05:00.463ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
38152025-03-02T05:05:00.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
38162025-03-02T05:05:00.465ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
38172025-03-02T05:05:00.465ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
38182025-03-02T05:05:00.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
38192025-03-02T05:05:00.474ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38202025-03-02T05:05:00.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38212025-03-02T05:05:00.475ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
38222025-03-02T05:05:00.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
38232025-03-02T05:05:00.478ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38242025-03-02T05:05:00.480ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38252025-03-02T05:05:00.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38262025-03-02T05:05:00.484ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
38272025-03-02T05:05:00.484ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
38282025-03-02T05:05:00.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
38292025-03-02T05:05:00.489ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
38302025-03-02T05:05:00.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
38312025-03-02T05:05:00.497ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
38322025-03-02T05:05:00.497ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
38332025-03-02T05:05:00.497ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
38342025-03-02T05:05:00.497ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 55.002596ms iteration = 55 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
38352025-03-02T05:05:00.499ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
38362025-03-02T05:05:00.499ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 57.143264ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
38372025-03-02T05:05:00.500ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3629, tv_nsec: 729542138 } })
38382025-03-02T05:05:00.500ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
38392025-03-02T05:05:00.501ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38402025-03-02T05:05:00.506ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
38412025-03-02T05:05:00.506ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38422025-03-02T05:05:00.506ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38432025-03-02T05:05:00.515ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
38442025-03-02T05:05:00.515ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
38452025-03-02T05:05:00.515ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
38462025-03-02T05:05:00.515ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
38472025-03-02T05:05:00.515ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38482025-03-02T05:05:00.516ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
38492025-03-02T05:05:00.516ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
38502025-03-02T05:05:00.516ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
38512025-03-02T05:05:00.516ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
38522025-03-02T05:05:00.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
38532025-03-02T05:05:00.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
38542025-03-02T05:05:00.519ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
38552025-03-02T05:05:00.519ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
38562025-03-02T05:05:00.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
38572025-03-02T05:05:00.521ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38582025-03-02T05:05:00.522ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38592025-03-02T05:05:00.523ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 83.396006ms iteration = 5 status = {"nsuccess":5}
38602025-03-02T05:05:00.523ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
38612025-03-02T05:05:00.523ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 84.324928ms iteration = 55 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
38622025-03-02T05:05:00.527ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
38632025-03-02T05:05:00.527ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
38642025-03-02T05:05:00.527ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
38652025-03-02T05:05:00.527ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
38662025-03-02T05:05:00.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
38672025-03-02T05:05:00.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
38682025-03-02T05:05:00.530ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
38692025-03-02T05:05:00.534ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
38702025-03-02T05:05:00.534ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 0 version = 2
38712025-03-02T05:05:00.534ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
38722025-03-02T05:05:00.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
38732025-03-02T05:05:00.537ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 93.620979ms iteration = 19 status = {"errors":[],"requests_created_ok":[]}
38742025-03-02T05:05:00.538ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
38752025-03-02T05:05:00.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
38762025-03-02T05:05:00.539ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38772025-03-02T05:05:00.540ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38782025-03-02T05:05:00.540ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38792025-03-02T05:05:00.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38802025-03-02T05:05:00.541ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
38812025-03-02T05:05:00.541ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
38822025-03-02T05:05:00.541ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
38832025-03-02T05:05:00.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
38842025-03-02T05:05:00.544ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
38852025-03-02T05:05:00.544ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
38862025-03-02T05:05:00.544ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
38872025-03-02T05:05:00.544ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
38882025-03-02T05:05:00.544ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
38892025-03-02T05:05:00.544ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
38902025-03-02T05:05:00.544ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
38912025-03-02T05:05:00.544ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 104.894396ms iteration = 10 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
38922025-03-02T05:05:00.544ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
38932025-03-02T05:05:00.545ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38942025-03-02T05:05:00.545ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38952025-03-02T05:05:00.545ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
38962025-03-02T05:05:00.545ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
38972025-03-02T05:05:00.546ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e6a8647-fd86-4f9c-977e-aa5085dc6fd8", "content-length": "26", "date": "Sun, 02 Mar 2025 05:05:00 GMT"} })
38982025-03-02T05:05:00.546ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
38992025-03-02T05:05:00.546ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39002025-03-02T05:05:00.547ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 1 version = 2
39012025-03-02T05:05:00.547ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
39022025-03-02T05:05:00.547ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-02 04:56:26.100923 UTC dns_group = external generation = 2
39032025-03-02T05:05:00.547ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 110.937905ms iteration = 10 status = {"generation":2}
39042025-03-02T05:05:00.548ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
39052025-03-02T05:05:00.548ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 108.524054ms iteration = 19 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
39062025-03-02T05:05:00.548ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
39072025-03-02T05:05:00.550ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39082025-03-02T05:05:00.551ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
39092025-03-02T05:05:00.551ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
39102025-03-02T05:05:00.551ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
39112025-03-02T05:05:00.553ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
39122025-03-02T05:05:00.553ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 110.501929ms iteration = 19 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
39132025-03-02T05:05:00.554ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
39142025-03-02T05:05:00.554ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
39152025-03-02T05:05:00.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
39162025-03-02T05:05:00.555ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 112.760729ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
39172025-03-02T05:05:00.555ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
39182025-03-02T05:05:00.555ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
39192025-03-02T05:05:00.555ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-02 04:56:25.487890 UTC dns_group = internal generation = 1
39202025-03-02T05:05:00.555ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 118.981969ms iteration = 10 status = {"generation":1}
39212025-03-02T05:05:00.557ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 118.041517ms iteration = 19 status = 0
39222025-03-02T05:05:00.558ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
39232025-03-02T05:05:00.558ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
39242025-03-02T05:05:00.558ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
39252025-03-02T05:05:00.559ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
39262025-03-02T05:05:00.559ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
39272025-03-02T05:05:00.562ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39282025-03-02T05:05:00.562ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39292025-03-02T05:05:00.563ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 120.354587ms iteration = 19 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
39302025-03-02T05:05:00.564ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39312025-03-02T05:05:00.566ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
39322025-03-02T05:05:00.566ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39332025-03-02T05:05:00.566ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
39342025-03-02T05:05:00.566ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
39352025-03-02T05:05:00.566ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
39362025-03-02T05:05:00.566ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
39372025-03-02T05:05:00.566ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
39382025-03-02T05:05:00.566ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
39392025-03-02T05:05:00.567ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
39402025-03-02T05:05:00.569ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
39412025-03-02T05:05:00.569ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
39422025-03-02T05:05:00.569ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 126.178353ms iteration = 19 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
39432025-03-02T05:05:00.576ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
39442025-03-02T05:05:00.577ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
39452025-03-02T05:05:00.597ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
39462025-03-02T05:05:00.597ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
39472025-03-02T05:05:00.597ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "63e85a90-423e-4ea1-85fc-1dd5aca3b31b", "content-length": "26", "date": "Sun, 02 Mar 2025 05:05:00 GMT"} })
39482025-03-02T05:05:00.597ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
39492025-03-02T05:05:00.600ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
39502025-03-02T05:05:00.622ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39512025-03-02T05:05:00.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39522025-03-02T05:05:00.625ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39532025-03-02T05:05:00.627ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-02T04:55:57.280988Z, time_modified: 2025-03-02T04:55:57.280988Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }) }
39542025-03-02T05:05:00.627ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 190.339429ms iteration = 10 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]}},"default_endpoint":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
39552025-03-02T05:05:00.627ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
39562025-03-02T05:05:00.627ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39572025-03-02T05:05:00.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39582025-03-02T05:05:00.631ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39592025-03-02T05:05:00.634ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
39602025-03-02T05:05:00.634ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39612025-03-02T05:05:00.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39622025-03-02T05:05:00.666ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
39632025-03-02T05:05:00.666ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 223.649623ms iteration = 10 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
39642025-03-02T05:05:00.684ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
39652025-03-02T05:05:00.684ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "d6cc4d3d-1946-4b4e-b72c-2ed9a169dd20", "date": "Sun, 02 Mar 2025 05:05:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
39662025-03-02T05:05:00.684ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
39672025-03-02T05:05:00.687ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "6b8d5898-5d61-4703-93a5-6d19fa93cf8b", "date": "Sun, 02 Mar 2025 05:05:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
39682025-03-02T05:05:00.687ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
39692025-03-02T05:05:00.689ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
39702025-03-02T05:05:00.689ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 252.672251ms iteration = 12 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
39712025-03-02T05:05:00.704ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
39722025-03-02T05:05:00.704ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "1711a164-987c-4a19-b470-b15a0f74f719", "content-length": "1", "date": "Sun, 02 Mar 2025 05:04:59 GMT"} })
39732025-03-02T05:05:00.705ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 265.412917ms iteration = 20 status = {"physical_disks_added":0}
39742025-03-02T05:05:00.706ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
39752025-03-02T05:05:00.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
39762025-03-02T05:05:00.708ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 271.718879ms iteration = 19 status = null
39772025-03-02T05:05:00.711ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
39782025-03-02T05:05:00.711ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
39792025-03-02T05:05:00.711ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab4a1f38-6104-4276-9062-59f177586bc3", "content-length": "26", "date": "Sun, 02 Mar 2025 05:05:00 GMT"} })
39802025-03-02T05:05:00.711ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
39812025-03-02T05:05:00.713ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
39822025-03-02T05:05:00.778ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
39832025-03-02T05:05:00.800ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
39842025-03-02T05:05:00.800ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "09929aa5-95a5-4107-a7c9-33d4cf66d84a", "content-length": "691", "date": "Sun, 02 Mar 2025 05:05:00 GMT"} })
39852025-03-02T05:05:00.800ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
39862025-03-02T05:05:00.800ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
39872025-03-02T05:05:00.800ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
39882025-03-02T05:05:00.803ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 360.285983ms iteration = 19 status = {}
39892025-03-02T05:05:00.813ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
39902025-03-02T05:05:00.813ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
39912025-03-02T05:05:00.813ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "117a951f-bd42-4728-9b8c-42982b4a076c", "content-length": "372", "date": "Sun, 02 Mar 2025 05:05:00 GMT"} })
39922025-03-02T05:05:00.813ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39932025-03-02T05:05:00.814ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
39942025-03-02T05:05:00.834ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
39952025-03-02T05:05:00.834ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
39962025-03-02T05:05:00.835ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "cdd30d9c-2feb-41bf-ba6c-6b497d14e379", "date": "Sun, 02 Mar 2025 05:05:00 GMT"} })
39972025-03-02T05:05:00.835ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39982025-03-02T05:05:00.835ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
39992025-03-02T05:05:00.837ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40002025-03-02T05:05:00.837ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40012025-03-02T05:05:00.841ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40022025-03-02T05:05:00.841ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40032025-03-02T05:05:00.843ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 407.042832ms iteration = 19 status = {}
40042025-03-02T05:05:00.844ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
40052025-03-02T05:05:00.844ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
40062025-03-02T05:05:00.844ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "da0c791a-1918-4766-9234-5f42fe05b90e", "date": "Sun, 02 Mar 2025 05:05:00 GMT"} })
40072025-03-02T05:05:00.846ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 403.82956ms iteration = 19 status = {}
40082025-03-02T05:05:00.884ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
40092025-03-02T05:05:00.885ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "bd8435d3-166c-4577-b93f-fe814ec2b5f5", "date": "Sun, 02 Mar 2025 05:05:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
40102025-03-02T05:05:00.885ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
40112025-03-02T05:05:00.886ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "2cd43c62-576b-4e2f-8c31-1a1174f4a145", "date": "Sun, 02 Mar 2025 05:05:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
40122025-03-02T05:05:00.886ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
40132025-03-02T05:05:00.887ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "d6f668f3-154b-4955-ac91-fc943d8f4813", "date": "Sun, 02 Mar 2025 05:05:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
40142025-03-02T05:05:00.887ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
40152025-03-02T05:05:00.888ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
40162025-03-02T05:05:00.888ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 452.1744ms iteration = 12 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
40172025-03-02T05:05:00.933ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40182025-03-02T05:05:00.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40192025-03-02T05:05:00.935ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
40202025-03-02T05:05:00.935ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40212025-03-02T05:05:00.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40222025-03-02T05:05:00.944ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
40232025-03-02T05:05:00.945ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4195f95-88fd-493c-93e7-57f22869b458", "content-length": "134", "date": "Sun, 02 Mar 2025 05:04:59 GMT"} })
40242025-03-02T05:05:00.945ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
40252025-03-02T05:05:00.945ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
40262025-03-02T05:05:00.945ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
40272025-03-02T05:05:00.945ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
40282025-03-02T05:05:00.945ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
40292025-03-02T05:05:00.945ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
40302025-03-02T05:05:00.945ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
40312025-03-02T05:05:00.945ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
40322025-03-02T05:05:00.945ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
40332025-03-02T05:05:00.945ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
40342025-03-02T05:05:00.945ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
40352025-03-02T05:05:00.945ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40362025-03-02T05:05:00.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40372025-03-02T05:05:00.947ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
40382025-03-02T05:05:00.947ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c4bc4b6-7b92-4e4c-a74c-e3076ad13131", "content-length": "2", "date": "Sun, 02 Mar 2025 05:04:59 GMT"} })
40392025-03-02T05:05:00.947ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
40402025-03-02T05:05:00.948ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b0ef094-07bb-4930-a176-38d6792c5291", "content-length": "52", "date": "Sun, 02 Mar 2025 05:04:59 GMT"} })
40412025-03-02T05:05:00.948ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
40422025-03-02T05:05:00.948ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
40432025-03-02T05:05:01.040ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "e35af16e-c322-41a0-a67f-975100353c93", "date": "Sun, 02 Mar 2025 05:05:00 GMT"} })
40442025-03-02T05:05:01.040ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
40452025-03-02T05:05:01.040ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
40462025-03-02T05:05:01.041ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "b19c502e-fa5d-45e6-abd5-ed0285e3cd0d", "content-length": "682", "date": "Sun, 02 Mar 2025 05:05:00 GMT"} })
40472025-03-02T05:05:01.041ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40482025-03-02T05:05:01.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40492025-03-02T05:05:01.045ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40502025-03-02T05:05:01.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40512025-03-02T05:05:01.048ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40522025-03-02T05:05:01.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40532025-03-02T05:05:01.050ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40542025-03-02T05:05:01.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40552025-03-02T05:05:01.057ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 614.94175ms iteration = 19 status = {}
40562025-03-02T05:05:02.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 423 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = de89ee8c-4984-41d7-9dcb-f17713f858b8 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
40572025-03-02T05:05:10.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 56 reason = Timeout
40582025-03-02T05:05:10.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40592025-03-02T05:05:10.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40602025-03-02T05:05:10.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40612025-03-02T05:05:10.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 56 reason = Timeout
40622025-03-02T05:05:10.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40632025-03-02T05:05:10.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40642025-03-02T05:05:10.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
40652025-03-02T05:05:10.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40662025-03-02T05:05:10.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40672025-03-02T05:05:10.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40682025-03-02T05:05:10.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40692025-03-02T05:05:10.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40702025-03-02T05:05:10.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
40712025-03-02T05:05:10.450ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40722025-03-02T05:05:10.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40732025-03-02T05:05:10.465ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
40742025-03-02T05:05:10.465ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.581273ms iteration = 56 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
40752025-03-02T05:05:10.480ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40762025-03-02T05:05:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40772025-03-02T05:05:10.510ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 66.82482ms iteration = 56 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
40782025-03-02T05:05:12.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 289 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = c10ae098-f135-45cd-8cd0-97fc1ce9a3a1 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
40792025-03-02T05:05:20.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 57 reason = Timeout
40802025-03-02T05:05:20.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40812025-03-02T05:05:20.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40822025-03-02T05:05:20.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40832025-03-02T05:05:20.442ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 57 reason = Timeout
40842025-03-02T05:05:20.442ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40852025-03-02T05:05:20.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40862025-03-02T05:05:20.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40872025-03-02T05:05:20.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40882025-03-02T05:05:20.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
40892025-03-02T05:05:20.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40902025-03-02T05:05:20.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40912025-03-02T05:05:20.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40922025-03-02T05:05:20.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40932025-03-02T05:05:20.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40942025-03-02T05:05:20.451ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
40952025-03-02T05:05:20.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40962025-03-02T05:05:20.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40972025-03-02T05:05:20.463ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
40982025-03-02T05:05:20.463ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.543064ms iteration = 57 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
40992025-03-02T05:05:20.475ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.568214ms iteration = 57 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
41002025-03-02T05:05:22.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 427 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 4f713eba-62d4-4fa4-80b1-09ec4359c46d response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
41012025-03-02T05:05:30.436ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 20 reason = Timeout
41022025-03-02T05:05:30.436ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 20 reason = Timeout
41032025-03-02T05:05:30.436ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
41042025-03-02T05:05:30.436ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 20 reason = Timeout
41052025-03-02T05:05:30.436ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
41062025-03-02T05:05:30.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
41072025-03-02T05:05:30.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
41082025-03-02T05:05:30.439ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3659, tv_nsec: 668608726 } })
41092025-03-02T05:05:30.439ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 21 reason = Timeout
41102025-03-02T05:05:30.439ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 58 reason = Timeout
41112025-03-02T05:05:30.439ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 20 reason = Timeout
41122025-03-02T05:05:30.439ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 20 reason = Timeout
41132025-03-02T05:05:30.439ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
41142025-03-02T05:05:30.439ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
41152025-03-02T05:05:30.439ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
41162025-03-02T05:05:30.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41172025-03-02T05:05:30.439ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
41182025-03-02T05:05:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
41192025-03-02T05:05:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
41202025-03-02T05:05:30.440ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3659, tv_nsec: 669085632 } })
41212025-03-02T05:05:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41222025-03-02T05:05:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
41232025-03-02T05:05:30.440ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
41242025-03-02T05:05:30.440ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.772369ms iteration = 20 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
41252025-03-02T05:05:30.442ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41262025-03-02T05:05:30.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41272025-03-02T05:05:30.442ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41282025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 58 reason = Timeout
41292025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41302025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41312025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 20 reason = Timeout
41322025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
41332025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 20 reason = Timeout
41342025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
41352025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 20 reason = Timeout
41362025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
41372025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 20 reason = Timeout
41382025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
41392025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
41402025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 20 reason = Timeout
41412025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
41422025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 20 reason = Timeout
41432025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41442025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 20 reason = Timeout
41452025-03-02T05:05:30.443ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
41462025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
41472025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 20 reason = Timeout
41482025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
41492025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 20 reason = Timeout
41502025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
41512025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 20 reason = Timeout
41522025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
41532025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
41542025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
41552025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
41562025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
41572025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
41582025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41592025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
41602025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
41612025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
41622025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
41632025-03-02T05:05:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
41642025-03-02T05:05:30.444ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 20 reason = Timeout
41652025-03-02T05:05:30.444ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
41662025-03-02T05:05:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
41672025-03-02T05:05:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
41682025-03-02T05:05:30.445ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
41692025-03-02T05:05:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
41702025-03-02T05:05:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
41712025-03-02T05:05:30.446ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
41722025-03-02T05:05:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
41732025-03-02T05:05:30.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41742025-03-02T05:05:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
41752025-03-02T05:05:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
41762025-03-02T05:05:30.446ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
41772025-03-02T05:05:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41782025-03-02T05:05:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
41792025-03-02T05:05:30.448ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41802025-03-02T05:05:30.449ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
41812025-03-02T05:05:30.449ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
41822025-03-02T05:05:30.449ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
41832025-03-02T05:05:30.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
41842025-03-02T05:05:30.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
41852025-03-02T05:05:30.450ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
41862025-03-02T05:05:30.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
41872025-03-02T05:05:30.450ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41882025-03-02T05:05:30.451ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
41892025-03-02T05:05:30.451ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 8.296358ms iteration = 20 status = {"errors":[],"garbage_collect_requested":[]}
41902025-03-02T05:05:30.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
41912025-03-02T05:05:30.452ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
41922025-03-02T05:05:30.452ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
41932025-03-02T05:05:30.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
41942025-03-02T05:05:30.454ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
41952025-03-02T05:05:30.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
41962025-03-02T05:05:30.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41972025-03-02T05:05:30.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41982025-03-02T05:05:30.456ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
41992025-03-02T05:05:30.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42002025-03-02T05:05:30.457ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 14.806962ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
42012025-03-02T05:05:30.458ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42022025-03-02T05:05:30.459ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 15.463251ms iteration = 20 status = {"errors":[],"requests_created_ok":[]}
42032025-03-02T05:05:30.460ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
42042025-03-02T05:05:30.460ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42052025-03-02T05:05:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42062025-03-02T05:05:30.466ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
42072025-03-02T05:05:30.466ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
42082025-03-02T05:05:30.466ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
42092025-03-02T05:05:30.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
42102025-03-02T05:05:30.500ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
42112025-03-02T05:05:30.501ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
42122025-03-02T05:05:30.501ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42132025-03-02T05:05:30.503ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
42142025-03-02T05:05:30.503ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42152025-03-02T05:05:30.503ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
42162025-03-02T05:05:30.503ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42172025-03-02T05:05:30.503ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
42182025-03-02T05:05:30.505ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 61.847255ms iteration = 20 status = {"errors":[],"finish_invoked_ok":[]}
42192025-03-02T05:05:30.505ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42202025-03-02T05:05:30.506ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3659, tv_nsec: 735695279 } })
42212025-03-02T05:05:30.507ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
42222025-03-02T05:05:30.508ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
42232025-03-02T05:05:30.508ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
42242025-03-02T05:05:30.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
42252025-03-02T05:05:30.511ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
42262025-03-02T05:05:30.512ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
42272025-03-02T05:05:30.513ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
42282025-03-02T05:05:30.513ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
42292025-03-02T05:05:30.514ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 71.101206ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
42302025-03-02T05:05:30.515ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
42312025-03-02T05:05:30.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
42322025-03-02T05:05:30.517ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 74.083445ms iteration = 20 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
42332025-03-02T05:05:30.518ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42342025-03-02T05:05:30.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42352025-03-02T05:05:30.520ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42362025-03-02T05:05:30.520ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
42372025-03-02T05:05:30.520ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
42382025-03-02T05:05:30.520ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 77.768483ms iteration = 20 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
42392025-03-02T05:05:30.521ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42402025-03-02T05:05:30.521ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
42412025-03-02T05:05:30.521ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 77.957655ms iteration = 20 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
42422025-03-02T05:05:30.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42432025-03-02T05:05:30.523ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42442025-03-02T05:05:30.524ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
42452025-03-02T05:05:30.524ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42462025-03-02T05:05:30.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42472025-03-02T05:05:30.526ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
42482025-03-02T05:05:30.526ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42492025-03-02T05:05:30.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42502025-03-02T05:05:30.527ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 88.123688ms iteration = 20 status = 0
42512025-03-02T05:05:30.529ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42522025-03-02T05:05:30.530ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
42532025-03-02T05:05:30.530ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
42542025-03-02T05:05:30.530ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d50ff36-df9b-493f-9cfb-de6111485d6c", "content-length": "26", "date": "Sun, 02 Mar 2025 05:05:30 GMT"} })
42552025-03-02T05:05:30.530ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
42562025-03-02T05:05:30.531ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
42572025-03-02T05:05:30.531ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
42582025-03-02T05:05:30.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
42592025-03-02T05:05:30.532ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
42602025-03-02T05:05:30.534ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
42612025-03-02T05:05:30.534ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 94.872305ms iteration = 58 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
42622025-03-02T05:05:30.536ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 93.372196ms iteration = 58 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
42632025-03-02T05:05:30.537ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
42642025-03-02T05:05:30.537ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
42652025-03-02T05:05:30.537ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3ffe54f-b7fc-405f-beb8-20424c32aade", "content-length": "26", "date": "Sun, 02 Mar 2025 05:05:30 GMT"} })
42662025-03-02T05:05:30.537ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
42672025-03-02T05:05:30.538ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
42682025-03-02T05:05:30.539ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
42692025-03-02T05:05:30.539ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 99.566587ms iteration = 20 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
42702025-03-02T05:05:30.564ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 124.824066ms iteration = 21 status = {"physical_disks_added":0}
42712025-03-02T05:05:30.577ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42722025-03-02T05:05:30.577ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42732025-03-02T05:05:30.585ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42742025-03-02T05:05:30.586ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42752025-03-02T05:05:30.587ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42762025-03-02T05:05:30.589ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
42772025-03-02T05:05:30.589ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42782025-03-02T05:05:30.589ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42792025-03-02T05:05:30.591ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42802025-03-02T05:05:30.593ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
42812025-03-02T05:05:30.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42822025-03-02T05:05:30.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42832025-03-02T05:05:30.647ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 210.935567ms iteration = 20 status = {}
42842025-03-02T05:05:30.655ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
42852025-03-02T05:05:30.655ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
42862025-03-02T05:05:30.656ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "195460f6-726c-41d9-9d63-bb301d5f9487", "content-length": "26", "date": "Sun, 02 Mar 2025 05:05:30 GMT"} })
42872025-03-02T05:05:30.656ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
42882025-03-02T05:05:30.657ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
42892025-03-02T05:05:30.659ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
42902025-03-02T05:05:30.660ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3fe41a0-8b48-44d3-abe1-634c3ad50357", "content-length": "1", "date": "Sun, 02 Mar 2025 05:05:30 GMT"} })
42912025-03-02T05:05:30.662ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
42922025-03-02T05:05:30.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
42932025-03-02T05:05:30.664ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 227.808257ms iteration = 20 status = null
42942025-03-02T05:05:30.706ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
42952025-03-02T05:05:30.706ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
42962025-03-02T05:05:30.706ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "f92d0be9-9568-44e8-b1b7-2c0ec946b712", "content-length": "372", "date": "Sun, 02 Mar 2025 05:05:30 GMT"} })
42972025-03-02T05:05:30.706ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42982025-03-02T05:05:30.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42992025-03-02T05:05:30.709ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
43002025-03-02T05:05:30.709ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e480c50-1faf-49fe-8850-a7bc7a9d5a3b", "content-length": "691", "date": "Sun, 02 Mar 2025 05:05:30 GMT"} })
43012025-03-02T05:05:30.709ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
43022025-03-02T05:05:30.709ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
43032025-03-02T05:05:30.709ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
43042025-03-02T05:05:30.711ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 268.305705ms iteration = 20 status = {}
43052025-03-02T05:05:30.720ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
43062025-03-02T05:05:30.720ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
43072025-03-02T05:05:30.720ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "28539950-1ded-45a1-8f0a-0c523e829430", "date": "Sun, 02 Mar 2025 05:05:30 GMT"} })
43082025-03-02T05:05:30.720ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43092025-03-02T05:05:30.721ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43102025-03-02T05:05:30.723ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43112025-03-02T05:05:30.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43122025-03-02T05:05:30.725ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43132025-03-02T05:05:30.726ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43142025-03-02T05:05:30.729ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
43152025-03-02T05:05:30.729ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
43162025-03-02T05:05:30.729ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "38a04041-0ab7-4202-9867-e486ee11c6ca", "date": "Sun, 02 Mar 2025 05:05:30 GMT"} })
43172025-03-02T05:05:30.730ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 287.751039ms iteration = 20 status = {}
43182025-03-02T05:05:30.781ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43192025-03-02T05:05:30.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43202025-03-02T05:05:30.784ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
43212025-03-02T05:05:30.784ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43222025-03-02T05:05:30.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43232025-03-02T05:05:30.793ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
43242025-03-02T05:05:30.794ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e265a05-4451-4528-88bc-87ca69dadf28", "content-length": "134", "date": "Sun, 02 Mar 2025 05:05:30 GMT"} })
43252025-03-02T05:05:30.794ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
43262025-03-02T05:05:30.794ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
43272025-03-02T05:05:30.794ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
43282025-03-02T05:05:30.794ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
43292025-03-02T05:05:30.794ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
43302025-03-02T05:05:30.794ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
43312025-03-02T05:05:30.794ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
43322025-03-02T05:05:30.794ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
43332025-03-02T05:05:30.794ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
43342025-03-02T05:05:30.794ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
43352025-03-02T05:05:30.794ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
43362025-03-02T05:05:30.794ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43372025-03-02T05:05:30.795ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43382025-03-02T05:05:30.796ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
43392025-03-02T05:05:30.796ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "071938c4-aa97-4cc6-ba39-e55934d846ea", "content-length": "2", "date": "Sun, 02 Mar 2025 05:05:30 GMT"} })
43402025-03-02T05:05:30.796ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
43412025-03-02T05:05:30.796ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "49acfd2b-52cb-4bcf-91f8-91f2b444f7a5", "content-length": "52", "date": "Sun, 02 Mar 2025 05:05:30 GMT"} })
43422025-03-02T05:05:30.796ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
43432025-03-02T05:05:30.796ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
43442025-03-02T05:05:30.919ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "02080ca5-a509-4c62-b7e9-ceb03eb277f3", "date": "Sun, 02 Mar 2025 05:05:30 GMT"} })
43452025-03-02T05:05:30.919ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
43462025-03-02T05:05:30.919ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
43472025-03-02T05:05:30.919ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "82df76f0-5c42-4c68-92c9-c8ab385c0d95", "content-length": "682", "date": "Sun, 02 Mar 2025 05:05:30 GMT"} })
43482025-03-02T05:05:30.919ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43492025-03-02T05:05:30.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43502025-03-02T05:05:30.923ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43512025-03-02T05:05:30.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43522025-03-02T05:05:30.927ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43532025-03-02T05:05:30.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43542025-03-02T05:05:30.929ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43552025-03-02T05:05:30.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43562025-03-02T05:05:30.936ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 493.372437ms iteration = 20 status = {}
43572025-03-02T05:05:32.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 302 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = a9cfd72a-5247-42b9-91a6-374f8bec312a response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
43582025-03-02T05:05:40.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 59 reason = Timeout
43592025-03-02T05:05:40.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43602025-03-02T05:05:40.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43612025-03-02T05:05:40.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43622025-03-02T05:05:40.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 59 reason = Timeout
43632025-03-02T05:05:40.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43642025-03-02T05:05:40.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43652025-03-02T05:05:40.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
43662025-03-02T05:05:40.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43672025-03-02T05:05:40.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43682025-03-02T05:05:40.445ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43692025-03-02T05:05:40.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43702025-03-02T05:05:40.447ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.403684ms iteration = 59 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
43712025-03-02T05:05:40.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43722025-03-02T05:05:40.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43732025-03-02T05:05:40.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43742025-03-02T05:05:40.451ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
43752025-03-02T05:05:40.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43762025-03-02T05:05:40.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43772025-03-02T05:05:40.462ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
43782025-03-02T05:05:40.462ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.833605ms iteration = 59 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
43792025-03-02T05:05:42.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 417 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 12bf2781-cb30-40a5-9f4f-d8f7cf5abbc0 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
43802025-03-02T05:05:50.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 60 reason = Timeout
43812025-03-02T05:05:50.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43822025-03-02T05:05:50.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43832025-03-02T05:05:50.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43842025-03-02T05:05:50.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 60 reason = Timeout
43852025-03-02T05:05:50.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43862025-03-02T05:05:50.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43872025-03-02T05:05:50.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
43882025-03-02T05:05:50.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43892025-03-02T05:05:50.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43902025-03-02T05:05:50.445ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43912025-03-02T05:05:50.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43922025-03-02T05:05:50.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43932025-03-02T05:05:50.447ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.625968ms iteration = 60 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
43942025-03-02T05:05:50.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43952025-03-02T05:05:50.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43962025-03-02T05:05:50.451ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
43972025-03-02T05:05:50.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43982025-03-02T05:05:50.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43992025-03-02T05:05:50.460ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
44002025-03-02T05:05:50.460ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.365672ms iteration = 60 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
44012025-03-02T05:05:52.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 262 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 4ae086ba-9753-4d7f-b505-2c14443c7c65 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
44022025-03-02T05:05:57.896ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:61257
44032025-03-02T05:05:57.896ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 86b2fe67-7fa5-4820-b275-e3b754cbe0b9 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
44042025-03-02T05:05:57.896ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 86b2fe67-7fa5-4820-b275-e3b754cbe0b9 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
44052025-03-02T05:05:57.898ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2619 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 86b2fe67-7fa5-4820-b275-e3b754cbe0b9 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
44062025-03-02T05:05:57.899ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 5d0b82f0-2c34-4dc6-8faf-e8d43c93c0e1 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
44072025-03-02T05:05:57.899ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 5d0b82f0-2c34-4dc6-8faf-e8d43c93c0e1 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
44082025-03-02T05:05:57.901ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2597 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 5d0b82f0-2c34-4dc6-8faf-e8d43c93c0e1 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
44092025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 13 reason = Timeout
44102025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 21 reason = Timeout
44112025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
44122025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 21 reason = Timeout
44132025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 11 reason = Timeout
44142025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 11 reason = Timeout
44152025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
44162025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
44172025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 11 reason = Timeout
44182025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 13 reason = Timeout
44192025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
44202025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
44212025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 11 reason = Timeout
44222025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 11 reason = Timeout
44232025-03-02T05:06:00.437ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-02 04:56:00.437329489 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
44242025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
44252025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 11 reason = Timeout
44262025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
44272025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 21 reason = Timeout
44282025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
44292025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
44302025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
44312025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
44322025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
44332025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
44342025-03-02T05:06:00.437ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3bf56dce-8353-46bb-a2f0-982fbd876e0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("742da26a-88eb-42ac-9329-e125fe787858.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 3689, tv_nsec: 666491639 } })
44352025-03-02T05:06:00.437ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4a50e69a-1735-4503-96cb-55c6cc854815.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 3689, tv_nsec: 666580500 } })
44362025-03-02T05:06:00.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
44372025-03-02T05:06:00.438ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
44382025-03-02T05:06:00.438ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 985.493\u{b5}s iteration = 11 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
44392025-03-02T05:06:00.438ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
44402025-03-02T05:06:00.438ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.079134ms iteration = 11 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
44412025-03-02T05:06:00.438ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
44422025-03-02T05:06:00.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
44432025-03-02T05:06:00.439ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
44442025-03-02T05:06:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
44452025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44462025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 22 reason = Timeout
44472025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
44482025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 11 reason = Timeout
44492025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 21 reason = Timeout
44502025-03-02T05:06:00.440ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
44512025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
44522025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
44532025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 21 reason = Timeout
44542025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
44552025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 61 reason = Timeout
44562025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 4 reason = Timeout
44572025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44582025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Timeout
44592025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
44602025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44612025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 12 reason = Timeout
44622025-03-02T05:06:00.440ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca
44632025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 8.88\u{b5}s iteration = 12 status = {"enabled":false,"target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca"}
44642025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
44652025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
44662025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
44672025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
44682025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44692025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
44702025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44712025-03-02T05:06:00.440ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.502806ms iteration = 21 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
44722025-03-02T05:06:00.441ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3689, tv_nsec: 670131397 } })
44732025-03-02T05:06:00.441ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
44742025-03-02T05:06:00.442ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3689, tv_nsec: 670890737 } })
44752025-03-02T05:06:00.442ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
44762025-03-02T05:06:00.442ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-02 04:56:00.437329489 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
44772025-03-02T05:06:00.442ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.214838ms iteration = 11 status = {"expiration":"2025-03-02T04:56:00.437329489Z","pruned":[]}
44782025-03-02T05:06:00.442ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
44792025-03-02T05:06:00.442ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
44802025-03-02T05:06:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
44812025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 21 reason = Timeout
44822025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 11 reason = Timeout
44832025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
44842025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
44852025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 21 reason = Timeout
44862025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
44872025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = saga_recovery iteration = 2 reason = Timeout
44882025-03-02T05:06:00.443ZDEBGnexus (SagaRecoverer): listing candidate sagas for recovery
44892025-03-02T05:06:00.443ZDEBGnexus (SagaRecoverer): roles roles = RoleSet { roles: {} }
44902025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 21 reason = Timeout
44912025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
44922025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 21 reason = Timeout
44932025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
44942025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 21 reason = Timeout
44952025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
44962025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
44972025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 21 reason = Timeout
44982025-03-02T05:06:00.443ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
44992025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45002025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 11 reason = Timeout
45012025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
45022025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 21 reason = Timeout
45032025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 11 reason = Timeout
45042025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
45052025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
45062025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 21 reason = Timeout
45072025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
45082025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 61 reason = Timeout
45092025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
45102025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 21 reason = Timeout
45112025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
45122025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 21 reason = Timeout
45132025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
45142025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 3 reason = Timeout
45152025-03-02T05:06:00.443ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
45162025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
45172025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
45182025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
45192025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
45202025-03-02T05:06:00.443ZDEBGnexus (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
45212025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
45222025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45232025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
45242025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
45252025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
45262025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
45272025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
45282025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
45292025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
45302025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
45312025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
45322025-03-02T05:06:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
45332025-03-02T05:06:00.444ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 21 reason = Timeout
45342025-03-02T05:06:00.444ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
45352025-03-02T05:06:00.444ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 3 reason = Timeout
45362025-03-02T05:06:00.444ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
45372025-03-02T05:06:00.444ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45382025-03-02T05:06:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
45392025-03-02T05:06:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
45402025-03-02T05:06:00.445ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45412025-03-02T05:06:00.445ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45422025-03-02T05:06:00.445ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45432025-03-02T05:06:00.446ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45442025-03-02T05:06:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
45452025-03-02T05:06:00.448ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
45462025-03-02T05:06:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
45472025-03-02T05:06:00.448ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
45482025-03-02T05:06:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
45492025-03-02T05:06:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
45502025-03-02T05:06:00.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
45512025-03-02T05:06:00.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
45522025-03-02T05:06:00.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
45532025-03-02T05:06:00.450ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
45542025-03-02T05:06:00.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
45552025-03-02T05:06:00.451ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45562025-03-02T05:06:00.451ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
45572025-03-02T05:06:00.451ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
45582025-03-02T05:06:00.451ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
45592025-03-02T05:06:00.451ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
45602025-03-02T05:06:00.451ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45612025-03-02T05:06:00.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
45622025-03-02T05:06:00.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45632025-03-02T05:06:00.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45642025-03-02T05:06:00.453ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
45652025-03-02T05:06:00.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
45662025-03-02T05:06:00.454ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
45672025-03-02T05:06:00.454ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
45682025-03-02T05:06:00.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
45692025-03-02T05:06:00.454ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
45702025-03-02T05:06:00.454ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45712025-03-02T05:06:00.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45722025-03-02T05:06:00.454ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
45732025-03-02T05:06:00.454ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45742025-03-02T05:06:00.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45752025-03-02T05:06:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
45762025-03-02T05:06:00.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
45772025-03-02T05:06:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
45782025-03-02T05:06:00.458ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 15.583203ms iteration = 21 status = {"errors":[],"finish_invoked_ok":[]}
45792025-03-02T05:06:00.463ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 19.785267ms iteration = 11 status = {"errors":[],"found_port_ok":[]}
45802025-03-02T05:06:00.466ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45812025-03-02T05:06:00.469ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
45822025-03-02T05:06:00.469ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45832025-03-02T05:06:00.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45842025-03-02T05:06:00.470ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
45852025-03-02T05:06:00.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
45862025-03-02T05:06:00.474ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45872025-03-02T05:06:00.476ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
45882025-03-02T05:06:00.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
45892025-03-02T05:06:00.477ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
45902025-03-02T05:06:00.477ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
45912025-03-02T05:06:00.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
45922025-03-02T05:06:00.477ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45932025-03-02T05:06:00.480ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
45942025-03-02T05:06:00.480ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
45952025-03-02T05:06:00.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
45962025-03-02T05:06:00.482ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45972025-03-02T05:06:00.483ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
45982025-03-02T05:06:00.484ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
45992025-03-02T05:06:00.487ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46002025-03-02T05:06:00.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46012025-03-02T05:06:00.489ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
46022025-03-02T05:06:00.489ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
46032025-03-02T05:06:00.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
46042025-03-02T05:06:00.489ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
46052025-03-02T05:06:00.489ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
46062025-03-02T05:06:00.490ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 46.589627ms iteration = 11 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
46072025-03-02T05:06:00.493ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46082025-03-02T05:06:00.493ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46092025-03-02T05:06:00.493ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46102025-03-02T05:06:00.494ZDEBGnexus (BackgroundTasks): inventory_prune_one: eligible for removal background_task = inventory_collection candidates = [(f9d9cc44-3890-4981-b01a-88d6d333d2a6, 0), (04fe6cab-7296-40bc-8c51-f01f263ecdc5, 0), (bdce7a86-b4f6-4172-828c-5cc76c61e422, 0), (c9c17d4b-65c0-4a48-afa2-43a0fe16c480, 0)] collection_id = f9d9cc44-3890-4981-b01a-88d6d333d2a6
46112025-03-02T05:06:00.494ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46122025-03-02T05:06:00.494ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
46132025-03-02T05:06:00.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46142025-03-02T05:06:00.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
46152025-03-02T05:06:00.496ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
46162025-03-02T05:06:00.496ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46172025-03-02T05:06:00.496ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46182025-03-02T05:06:00.498ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46192025-03-02T05:06:00.501ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
46202025-03-02T05:06:00.501ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46212025-03-02T05:06:00.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46222025-03-02T05:06:00.509ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
46232025-03-02T05:06:00.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
46242025-03-02T05:06:00.520ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
46252025-03-02T05:06:00.520ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
46262025-03-02T05:06:00.520ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
46272025-03-02T05:06:00.520ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 77.459038ms iteration = 21 status = {"errors":[],"garbage_collect_requested":[]}
46282025-03-02T05:06:00.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
46292025-03-02T05:06:00.523ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46302025-03-02T05:06:00.526ZINFOnexus (SagaRecoverer): listed in-progress sagas count = 0 file = nexus/src/app/background/tasks/saga_recovery.rs:438
46312025-03-02T05:06:00.526ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
46322025-03-02T05:06:00.526ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
46332025-03-02T05:06:00.526ZDEBGnexus (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 83.137593ms iteration = 2 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
46342025-03-02T05:06:00.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
46352025-03-02T05:06:00.528ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
46362025-03-02T05:06:00.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
46372025-03-02T05:06:00.534ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46382025-03-02T05:06:00.537ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
46392025-03-02T05:06:00.537ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
46402025-03-02T05:06:00.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
46412025-03-02T05:06:00.544ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
46422025-03-02T05:06:00.544ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
46432025-03-02T05:06:00.545ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46442025-03-02T05:06:00.548ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
46452025-03-02T05:06:00.548ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46462025-03-02T05:06:00.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46472025-03-02T05:06:00.549ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
46482025-03-02T05:06:00.549ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
46492025-03-02T05:06:00.549ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
46502025-03-02T05:06:00.549ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
46512025-03-02T05:06:00.550ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
46522025-03-02T05:06:00.550ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
46532025-03-02T05:06:00.550ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
46542025-03-02T05:06:00.551ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46552025-03-02T05:06:00.551ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
46562025-03-02T05:06:00.551ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
46572025-03-02T05:06:00.551ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 110.971126ms iteration = 11 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
46582025-03-02T05:06:00.552ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
46592025-03-02T05:06:00.552ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
46602025-03-02T05:06:00.552ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
46612025-03-02T05:06:00.552ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
46622025-03-02T05:06:00.555ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
46632025-03-02T05:06:00.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
46642025-03-02T05:06:00.556ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
46652025-03-02T05:06:00.556ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
46662025-03-02T05:06:00.556ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
46672025-03-02T05:06:00.560ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 116.070942ms iteration = 21 status = {"errors":[],"requests_created_ok":[]}
46682025-03-02T05:06:00.562ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 121.761086ms iteration = 22 status = {"physical_disks_added":0}
46692025-03-02T05:06:00.562ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
46702025-03-02T05:06:00.563ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 122.740168ms iteration = 61 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
46712025-03-02T05:06:00.568ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
46722025-03-02T05:06:00.568ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
46732025-03-02T05:06:00.571ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
46742025-03-02T05:06:00.571ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 128.520994ms iteration = 21 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
46752025-03-02T05:06:00.572ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 129.070402ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
46762025-03-02T05:06:00.573ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
46772025-03-02T05:06:00.573ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
46782025-03-02T05:06:00.573ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
46792025-03-02T05:06:00.574ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46802025-03-02T05:06:00.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
46812025-03-02T05:06:00.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46822025-03-02T05:06:00.584ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
46832025-03-02T05:06:00.584ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
46842025-03-02T05:06:00.584ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "24b53d14-df82-417d-85be-ede4ba583ca5", "content-length": "26", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
46852025-03-02T05:06:00.584ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
46862025-03-02T05:06:00.589ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
46872025-03-02T05:06:00.592ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
46882025-03-02T05:06:00.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
46892025-03-02T05:06:00.596ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
46902025-03-02T05:06:00.596ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
46912025-03-02T05:06:00.596ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
46922025-03-02T05:06:00.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
46932025-03-02T05:06:00.596ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
46942025-03-02T05:06:00.598ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
46952025-03-02T05:06:00.599ZINFOnexus (BackgroundTasks): removed inventory collection background_task = inventory_collection collection_id = f9d9cc44-3890-4981-b01a-88d6d333d2a6 file = nexus/db-queries/src/db/datastore/inventory.rs:1442 ncabooses = 12 nclickhouse_keeper_membership = 0 ncollections = 1 ndatasets = 0 nerrors = 0 nnics = 0 nnvme_disk_firmware = 0 nphysical_disks = 0 nrot_pages = 8 nrots = 2 nsled_agent_zones = 0 nsled_agents = 0 nsps = 2 nzones = 0 nzpools = 0
46962025-03-02T05:06:00.599ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46972025-03-02T05:06:00.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46982025-03-02T05:06:00.600ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 156.67335ms iteration = 61 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
46992025-03-02T05:06:00.606ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
47002025-03-02T05:06:00.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
47012025-03-02T05:06:00.619ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
47022025-03-02T05:06:00.619ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
47032025-03-02T05:06:00.619ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "05b2ffc8-9e10-4eff-b5dd-5158d34535df", "content-length": "26", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
47042025-03-02T05:06:00.619ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
47052025-03-02T05:06:00.621ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
47062025-03-02T05:06:00.692ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
47072025-03-02T05:06:00.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
47082025-03-02T05:06:00.695ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47092025-03-02T05:06:00.695ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47102025-03-02T05:06:00.695ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3689, tv_nsec: 924573371 } })
47112025-03-02T05:06:00.696ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
47122025-03-02T05:06:00.697ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
47132025-03-02T05:06:00.697ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
47142025-03-02T05:06:00.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
47152025-03-02T05:06:00.699ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
47162025-03-02T05:06:00.699ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
47172025-03-02T05:06:00.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
47182025-03-02T05:06:00.739ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
47192025-03-02T05:06:00.754ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 316.854997ms iteration = 21 status = {}
47202025-03-02T05:06:00.755ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
47212025-03-02T05:06:00.755ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "5df03690-3e98-42f3-9502-9d07b3c028b3", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
47222025-03-02T05:06:00.755ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
47232025-03-02T05:06:00.757ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "4108f45b-dbf5-4f88-a6bd-c5e100362ebe", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
47242025-03-02T05:06:00.757ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
47252025-03-02T05:06:00.762ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
47262025-03-02T05:06:00.763ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 325.704582ms iteration = 13 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
47272025-03-02T05:06:00.777ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
47282025-03-02T05:06:00.778ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "586b9e84-879e-4270-8299-248bbac94c86", "content-length": "1", "date": "Sun, 02 Mar 2025 05:05:59 GMT"} })
47292025-03-02T05:06:00.779ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
47302025-03-02T05:06:00.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
47312025-03-02T05:06:00.798ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
47322025-03-02T05:06:00.798ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
47332025-03-02T05:06:00.798ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "69d03f20-51c7-40ea-b922-f3fe49fede0b", "content-length": "372", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
47342025-03-02T05:06:00.798ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
47352025-03-02T05:06:00.799ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
47362025-03-02T05:06:00.800ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47372025-03-02T05:06:00.800ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47382025-03-02T05:06:00.802ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47392025-03-02T05:06:00.804ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
47402025-03-02T05:06:00.804ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
47412025-03-02T05:06:00.807ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
47422025-03-02T05:06:00.807ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
47432025-03-02T05:06:00.807ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
47442025-03-02T05:06:00.810ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
47452025-03-02T05:06:00.810ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
47462025-03-02T05:06:00.810ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e46a2617-4903-4df2-b754-9e79850a4a21", "content-length": "26", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
47472025-03-02T05:06:00.810ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
47482025-03-02T05:06:00.811ZDEBGnexus (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(04fe6cab-7296-40bc-8c51-f01f263ecdc5, 0), (bdce7a86-b4f6-4172-828c-5cc76c61e422, 0), (c9c17d4b-65c0-4a48-afa2-43a0fe16c480, 0)]
47492025-03-02T05:06:00.812ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3690, tv_nsec: 40830535 } })
47502025-03-02T05:06:00.812ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
47512025-03-02T05:06:00.812ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
47522025-03-02T05:06:00.812ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 0 version = 2
47532025-03-02T05:06:00.812ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
47542025-03-02T05:06:00.813ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
47552025-03-02T05:06:00.813ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 376.231221ms iteration = 21 status = null
47562025-03-02T05:06:00.815ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
47572025-03-02T05:06:00.815ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
47582025-03-02T05:06:00.815ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-02 04:56:25.487890 UTC dns_group = internal generation = 1
47592025-03-02T05:06:00.815ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 378.588091ms iteration = 11 status = {"generation":1}
47602025-03-02T05:06:00.849ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 405.710254ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
47612025-03-02T05:06:00.850ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
47622025-03-02T05:06:00.850ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
47632025-03-02T05:06:00.856ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
47642025-03-02T05:06:00.856ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 416.559726ms iteration = 21 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
47652025-03-02T05:06:00.857ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47662025-03-02T05:06:00.859ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
47672025-03-02T05:06:00.859ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
47682025-03-02T05:06:00.859ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
47692025-03-02T05:06:00.859ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
47702025-03-02T05:06:00.859ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 416.605737ms iteration = 21 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
47712025-03-02T05:06:00.859ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
47722025-03-02T05:06:00.860ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47732025-03-02T05:06:00.861ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47742025-03-02T05:06:00.861ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47752025-03-02T05:06:00.863ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47762025-03-02T05:06:00.863ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47772025-03-02T05:06:00.863ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
47782025-03-02T05:06:00.863ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
47792025-03-02T05:06:00.863ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47802025-03-02T05:06:00.863ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
47812025-03-02T05:06:00.865ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
47822025-03-02T05:06:00.865ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
47832025-03-02T05:06:00.866ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
47842025-03-02T05:06:00.866ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
47852025-03-02T05:06:00.866ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
47862025-03-02T05:06:00.866ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 423.444326ms iteration = 21 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
47872025-03-02T05:06:00.867ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47882025-03-02T05:06:00.867ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
47892025-03-02T05:06:00.868ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "532e212f-47de-4aae-bdb9-fb05e174155e", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
47902025-03-02T05:06:00.868ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
47912025-03-02T05:06:00.868ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
47922025-03-02T05:06:00.869ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
47932025-03-02T05:06:00.869ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
47942025-03-02T05:06:00.869ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
47952025-03-02T05:06:00.869ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47962025-03-02T05:06:00.869ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
47972025-03-02T05:06:00.869ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "77a59b6c-6562-4dfb-a82e-3966eed4f11b", "content-length": "2", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
47982025-03-02T05:06:00.869ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:619 sled = http://[fd00:1122:3344:101::1]:12345
47992025-03-02T05:06:00.869ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
48002025-03-02T05:06:00.870ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
48012025-03-02T05:06:00.870ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
48022025-03-02T05:06:00.870ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
48032025-03-02T05:06:00.870ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "351f7b59-09b7-4756-80bb-85c2f5b21542", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
48042025-03-02T05:06:00.870ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
48052025-03-02T05:06:00.870ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48062025-03-02T05:06:00.871ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 426.522456ms iteration = 3 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":2,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"operation":"list","target_sled":"fee17c45-a85b-405f-9e58-2ca0ab03dea9","time":"2025-03-02T05:01:01.051764014Z"},{"operation":"list","target_sled":"fee17c45-a85b-405f-9e58-2ca0ab03dea9","time":"2025-03-02T05:06:00.869287966Z"}]}
48072025-03-02T05:06:00.871ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 1 version = 2
48082025-03-02T05:06:00.871ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
48092025-03-02T05:06:00.871ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-02 04:56:26.100923 UTC dns_group = external generation = 2
48102025-03-02T05:06:00.871ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 434.177685ms iteration = 11 status = {"generation":2}
48112025-03-02T05:06:00.871ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
48122025-03-02T05:06:00.871ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
48132025-03-02T05:06:00.871ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
48142025-03-02T05:06:00.871ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "0d40b732-5960-4345-9bc0-2c5a147b0f98", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48152025-03-02T05:06:00.871ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48162025-03-02T05:06:00.871ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
48172025-03-02T05:06:00.872ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48182025-03-02T05:06:00.872ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 432.284931ms iteration = 21 status = 0
48192025-03-02T05:06:00.872ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "9f442e56-09eb-41b1-9241-2d26b3f37dd2", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
48202025-03-02T05:06:00.872ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
48212025-03-02T05:06:00.873ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
48222025-03-02T05:06:00.873ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
48232025-03-02T05:06:00.873ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
48242025-03-02T05:06:00.873ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
48252025-03-02T05:06:00.873ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
48262025-03-02T05:06:00.874ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48272025-03-02T05:06:00.874ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
48282025-03-02T05:06:00.874ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 437.468478ms iteration = 13 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
48292025-03-02T05:06:00.874ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48302025-03-02T05:06:00.877ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48312025-03-02T05:06:00.878ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48322025-03-02T05:06:00.881ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
48332025-03-02T05:06:00.881ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
48342025-03-02T05:06:00.881ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "9738dfd2-27c8-47f4-b9f3-5cbf90ba2b36", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48352025-03-02T05:06:00.883ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 440.000001ms iteration = 21 status = {}
48362025-03-02T05:06:00.891ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48372025-03-02T05:06:00.893ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
48382025-03-02T05:06:00.893ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
48392025-03-02T05:06:00.893ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
48402025-03-02T05:06:00.909ZDEBGnexus (BackgroundTasks): begin collection background_task = inventory_collection
48412025-03-02T05:06:00.909ZDEBGnexus (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[fd00:1122:3344:101::2]:12225
48422025-03-02T05:06:00.909ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/ignition
48432025-03-02T05:06:00.909ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dce6fb0-8de3-4a6f-999e-f6b9a82c3dcd", "content-length": "476", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48442025-03-02T05:06:00.909ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0
48452025-03-02T05:06:00.909ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "562dfad9-2737-4e0f-8438-928d1d4e5d0f", "content-length": "734", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48462025-03-02T05:06:00.910ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0
48472025-03-02T05:06:00.910ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "000aa4ed-1814-4879-b658-206771fba213", "content-length": "111", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48482025-03-02T05:06:00.910ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1
48492025-03-02T05:06:00.910ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "85b03d0f-615b-4a25-ab8f-756061f44c59", "content-length": "111", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48502025-03-02T05:06:00.910ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0
48512025-03-02T05:06:00.910ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d64cdf51-cb21-45db-b98d-3fae38eac957", "content-length": "167", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48522025-03-02T05:06:00.910ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1
48532025-03-02T05:06:00.911ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dacbf471-175c-4970-9a88-d9f9b6d40abf", "content-length": "167", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48542025-03-02T05:06:00.911ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0
48552025-03-02T05:06:00.911ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca2006b6-f5fc-4fa5-b655-47fa032beb64", "content-length": "175", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48562025-03-02T05:06:00.911ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1
48572025-03-02T05:06:00.911ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c19b508-c123-4c99-892d-9db3c459ee48", "content-length": "175", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48582025-03-02T05:06:00.911ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa
48592025-03-02T05:06:00.911ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82ad7a62-159b-482d-8640-f21ede4b9134", "content-length": "702", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48602025-03-02T05:06:00.911ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
48612025-03-02T05:06:00.911ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e69320d-b951-4e09-802c-63f24b10a17c", "content-length": "727", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48622025-03-02T05:06:00.912ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
48632025-03-02T05:06:00.912ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f8faccd-db7c-4b7e-88cd-2b33d31e909a", "content-length": "729", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48642025-03-02T05:06:00.912ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
48652025-03-02T05:06:00.912ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63d24b8f-56b8-4a62-9fa8-6362ab075ce4", "content-length": "728", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48662025-03-02T05:06:00.912ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0
48672025-03-02T05:06:00.912ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b55c858-841b-49a7-ad95-78ead9d1c7dd", "content-length": "722", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48682025-03-02T05:06:00.912ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0
48692025-03-02T05:06:00.912ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fee94e1f-8949-4777-9245-779bae7d7844", "content-length": "109", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48702025-03-02T05:06:00.912ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1
48712025-03-02T05:06:00.913ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "efb997f4-16e9-47db-a1ec-a55276cbd7a3", "content-length": "109", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48722025-03-02T05:06:00.913ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0
48732025-03-02T05:06:00.913ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fffd2a2-c43e-4799-84e7-79c2b2d7d32c", "content-length": "169", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48742025-03-02T05:06:00.913ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1
48752025-03-02T05:06:00.913ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8336a577-08ae-4f80-a18a-d6964109f3e1", "content-length": "169", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48762025-03-02T05:06:00.913ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0
48772025-03-02T05:06:00.914ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "196a0135-79b1-43ef-86b8-94c2b4029b2c", "content-length": "178", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48782025-03-02T05:06:00.914ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1
48792025-03-02T05:06:00.914ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "715da451-20be-47a0-a85d-b4255b099399", "content-length": "178", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48802025-03-02T05:06:00.914ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa
48812025-03-02T05:06:00.914ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e400fb1a-79cc-469b-974a-02d72d7fa219", "content-length": "702", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48822025-03-02T05:06:00.914ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
48832025-03-02T05:06:00.914ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cfbcdc3-aa10-4549-b1e3-668cdfe4276f", "content-length": "727", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48842025-03-02T05:06:00.914ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
48852025-03-02T05:06:00.914ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c171534-e6ac-4394-9780-a8a6a8e33e11", "content-length": "729", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48862025-03-02T05:06:00.914ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
48872025-03-02T05:06:00.914ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67a02275-faba-4a62-b674-6b397c89e2ea", "content-length": "728", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
48882025-03-02T05:06:00.914ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48892025-03-02T05:06:00.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48902025-03-02T05:06:00.917ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48912025-03-02T05:06:00.918ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48922025-03-02T05:06:00.919ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48932025-03-02T05:06:00.919ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48942025-03-02T05:06:00.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48952025-03-02T05:06:00.920ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
48962025-03-02T05:06:00.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
48972025-03-02T05:06:00.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
48982025-03-02T05:06:00.921ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48992025-03-02T05:06:00.923ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
49002025-03-02T05:06:00.923ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 482.852789ms iteration = 4 status = {"blueprint_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","inventory_collection_id":"c9c17d4b-65c0-4a48-afa2-43a0fe16c480","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
49012025-03-02T05:06:00.925ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
49022025-03-02T05:06:00.925ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49032025-03-02T05:06:00.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49042025-03-02T05:06:00.926ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
49052025-03-02T05:06:00.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
49062025-03-02T05:06:00.928ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49072025-03-02T05:06:00.928ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49082025-03-02T05:06:00.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49092025-03-02T05:06:00.929ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49102025-03-02T05:06:00.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49112025-03-02T05:06:00.930ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
49122025-03-02T05:06:00.930ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
49132025-03-02T05:06:00.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
49142025-03-02T05:06:00.931ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49152025-03-02T05:06:00.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49162025-03-02T05:06:00.932ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49172025-03-02T05:06:00.932ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
49182025-03-02T05:06:00.932ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 488.737327ms iteration = 11 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
49192025-03-02T05:06:00.933ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
49202025-03-02T05:06:00.933ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
49212025-03-02T05:06:00.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
49222025-03-02T05:06:00.933ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49232025-03-02T05:06:00.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49242025-03-02T05:06:00.935ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49252025-03-02T05:06:00.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49262025-03-02T05:06:00.936ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49272025-03-02T05:06:00.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49282025-03-02T05:06:00.938ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-02T04:55:57.280988Z, time_modified: 2025-03-02T04:55:57.280988Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }) }
49292025-03-02T05:06:00.938ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 500.952346ms iteration = 11 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]}},"default_endpoint":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
49302025-03-02T05:06:00.939ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49312025-03-02T05:06:00.944ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
49322025-03-02T05:06:00.944ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49332025-03-02T05:06:00.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49342025-03-02T05:06:00.946ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49352025-03-02T05:06:00.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49362025-03-02T05:06:00.948ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49372025-03-02T05:06:00.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49382025-03-02T05:06:00.950ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49392025-03-02T05:06:00.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49402025-03-02T05:06:00.952ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49412025-03-02T05:06:00.952ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49422025-03-02T05:06:00.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49432025-03-02T05:06:00.953ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
49442025-03-02T05:06:00.953ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49452025-03-02T05:06:00.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49462025-03-02T05:06:00.955ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49472025-03-02T05:06:00.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49482025-03-02T05:06:00.966ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49492025-03-02T05:06:00.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49502025-03-02T05:06:00.967ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
49512025-03-02T05:06:00.967ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49522025-03-02T05:06:00.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49532025-03-02T05:06:00.977ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
49542025-03-02T05:06:00.977ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a20a6b1-3ca1-4b90-9b6c-b62f70598963", "content-length": "134", "date": "Sun, 02 Mar 2025 05:05:59 GMT"} })
49552025-03-02T05:06:00.977ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
49562025-03-02T05:06:00.977ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
49572025-03-02T05:06:00.977ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
49582025-03-02T05:06:00.977ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
49592025-03-02T05:06:00.977ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
49602025-03-02T05:06:00.977ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
49612025-03-02T05:06:00.977ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
49622025-03-02T05:06:00.977ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
49632025-03-02T05:06:00.978ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
49642025-03-02T05:06:00.978ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
49652025-03-02T05:06:00.978ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
49662025-03-02T05:06:00.978ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49672025-03-02T05:06:00.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49682025-03-02T05:06:00.980ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
49692025-03-02T05:06:00.980ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7636696-8e6e-4a4a-9848-80bde30932bf", "content-length": "2", "date": "Sun, 02 Mar 2025 05:05:59 GMT"} })
49702025-03-02T05:06:00.980ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
49712025-03-02T05:06:00.981ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "88a10df4-2a1a-49f0-b2d1-d859a5ee9142", "content-length": "52", "date": "Sun, 02 Mar 2025 05:05:59 GMT"} })
49722025-03-02T05:06:00.981ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
49732025-03-02T05:06:00.981ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
49742025-03-02T05:06:00.985ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49752025-03-02T05:06:00.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49762025-03-02T05:06:00.986ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49772025-03-02T05:06:00.992ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
49782025-03-02T05:06:00.993ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49792025-03-02T05:06:00.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49802025-03-02T05:06:00.994ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49812025-03-02T05:06:00.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49822025-03-02T05:06:01.026ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49832025-03-02T05:06:01.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49842025-03-02T05:06:01.028ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49852025-03-02T05:06:01.032ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
49862025-03-02T05:06:01.032ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49872025-03-02T05:06:01.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49882025-03-02T05:06:01.041ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
49892025-03-02T05:06:01.045ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49902025-03-02T05:06:01.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49912025-03-02T05:06:01.047ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49922025-03-02T05:06:01.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49932025-03-02T05:06:01.049ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49942025-03-02T05:06:01.049ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49952025-03-02T05:06:01.062ZDEBGnexus (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[fd00:1122:3344:101::1]:12345
49962025-03-02T05:06:01.062ZDEBGnexus (BackgroundTasks): client request SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/inventory
49972025-03-02T05:06:01.065ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49982025-03-02T05:06:01.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49992025-03-02T05:06:01.073ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50002025-03-02T05:06:01.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
50012025-03-02T05:06:01.075ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50022025-03-02T05:06:01.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
50032025-03-02T05:06:01.077ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50042025-03-02T05:06:01.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
50052025-03-02T05:06:01.078ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50062025-03-02T05:06:01.082ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
50072025-03-02T05:06:01.082ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50082025-03-02T05:06:01.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
50092025-03-02T05:06:01.083ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50102025-03-02T05:06:01.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
50112025-03-02T05:06:01.085ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50122025-03-02T05:06:01.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
50132025-03-02T05:06:01.086ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "be3cf565-865c-4dbc-8917-b0f230f4913c", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
50142025-03-02T05:06:01.086ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
50152025-03-02T05:06:01.086ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
50162025-03-02T05:06:01.086ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50172025-03-02T05:06:01.087ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "0eb812f1-78d8-47eb-9f89-20a85846d37c", "content-length": "682", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
50182025-03-02T05:06:01.087ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50192025-03-02T05:06:01.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50202025-03-02T05:06:01.087ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc20c94a-1675-4758-b60c-d6057b3fad85", "content-length": "691", "date": "Sun, 02 Mar 2025 05:06:01 GMT"} })
50212025-03-02T05:06:01.090ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
50222025-03-02T05:06:01.090ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
50232025-03-02T05:06:01.090ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
50242025-03-02T05:06:01.091ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 648.591309ms iteration = 21 status = {}
50252025-03-02T05:06:01.094ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
50262025-03-02T05:06:01.094ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50272025-03-02T05:06:01.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
50282025-03-02T05:06:01.094ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50292025-03-02T05:06:01.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50302025-03-02T05:06:01.098ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50312025-03-02T05:06:01.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50322025-03-02T05:06:01.100ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50332025-03-02T05:06:01.100ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50342025-03-02T05:06:01.101ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 4d1ce337-a06d-4661-8223-cff03228021a, kind: Service { id: aebed526-5e03-4a58-86b5-1baf17c88392 }, name: Name("nexus-aebed526-5e03-4a58-86b5-1baf17c88392"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 2])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7854a1af-bcd2-41ab-9d60-4bc505686339, kind: Service { id: 091811f5-b683-4991-98f0-82afb4d1f635 }, name: Name("nexus-091811f5-b683-4991-98f0-82afb4d1f635"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b52288c2-a03b-488b-890b-db71040a1e60, kind: Service { id: feddb706-b221-4d1f-920c-1d1979b5460b }, name: Name("nexus-feddb706-b221-4d1f-920c-1d1979b5460b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 242, 53])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 3e4064c5-fe59-43d0-88e0-3ed3e0a5eba7, kind: Service { id: c54e30d8-f5a2-495b-b2d8-a78319addcd7 }, name: Name("external-dns-c54e30d8-f5a2-495b-b2d8-a78319addcd7"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 198, 192])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ef033b7d-0ede-4639-ab7f-6d87d9d01240, kind: Service { id: 4a50e69a-1735-4503-96cb-55c6cc854815 }, name: Name("external-dns-4a50e69a-1735-4503-96cb-55c6cc854815"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 237, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
50352025-03-02T05:06:01.101ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50362025-03-02T05:06:01.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
50372025-03-02T05:06:01.103ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50382025-03-02T05:06:01.105ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
50392025-03-02T05:06:01.105ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50402025-03-02T05:06:01.105ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
50412025-03-02T05:06:01.107ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
50422025-03-02T05:06:01.107ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 3e4064c5-fe59-43d0-88e0-3ed3e0a5eba7, kind: Service { id: c54e30d8-f5a2-495b-b2d8-a78319addcd7 }, name: Name("external-dns-c54e30d8-f5a2-495b-b2d8-a78319addcd7"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 198, 192])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ef033b7d-0ede-4639-ab7f-6d87d9d01240, kind: Service { id: 4a50e69a-1735-4503-96cb-55c6cc854815 }, name: Name("external-dns-4a50e69a-1735-4503-96cb-55c6cc854815"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 237, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 4d1ce337-a06d-4661-8223-cff03228021a, kind: Service { id: aebed526-5e03-4a58-86b5-1baf17c88392 }, name: Name("nexus-aebed526-5e03-4a58-86b5-1baf17c88392"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 2])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7854a1af-bcd2-41ab-9d60-4bc505686339, kind: Service { id: 091811f5-b683-4991-98f0-82afb4d1f635 }, name: Name("nexus-091811f5-b683-4991-98f0-82afb4d1f635"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 141, 180])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b52288c2-a03b-488b-890b-db71040a1e60, kind: Service { id: feddb706-b221-4d1f-920c-1d1979b5460b }, name: Name("nexus-feddb706-b221-4d1f-920c-1d1979b5460b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 242, 53])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
50432025-03-02T05:06:01.107ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
50442025-03-02T05:06:01.108ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 665.212175ms iteration = 21 status = {}
50452025-03-02T05:06:01.239ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: fee17c45-a85b-405f-9e58-2ca0ab03dea9, time_created: 2025-03-02T04:56:21.866513Z, time_modified: 2025-03-02T04:56:21.866513Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: d6094382-b327-4998-b53d-b2d98a53005b, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
50462025-03-02T05:06:01.239ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
50472025-03-02T05:06:01.239ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50482025-03-02T05:06:01.240ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
50492025-03-02T05:06:01.242ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50502025-03-02T05:06:01.242ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
50512025-03-02T05:06:01.244ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50522025-03-02T05:06:01.246ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
50532025-03-02T05:06:01.278ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
50542025-03-02T05:06:01.288ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "89f56c7e-86f5-4d91-ba8f-de05cbe85a17", "date": "Sun, 02 Mar 2025 05:06:01 GMT"} })
50552025-03-02T05:06:01.289ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
50562025-03-02T05:06:01.289ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 845.650355ms
50572025-03-02T05:06:01.289ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 845.660955ms iteration = 3 status = {}
50582025-03-02T05:06:01.413ZDEBGnexus (BackgroundTasks): client response SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "07aa0f38-694e-4a53-a60e-643bb7b42a41", "content-length": "31973", "date": "Sun, 02 Mar 2025 05:06:00 GMT"} })
50592025-03-02T05:06:01.447ZDEBGnexus (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
50602025-03-02T05:06:01.447ZDEBGnexus (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
50612025-03-02T05:06:01.447ZDEBGnexus (BackgroundTasks): finished collection background_task = inventory_collection
50622025-03-02T05:06:01.448ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50632025-03-02T05:06:01.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50642025-03-02T05:06:01.450ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50652025-03-02T05:06:01.452ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
50662025-03-02T05:06:01.452ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50672025-03-02T05:06:01.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50682025-03-02T05:06:01.641ZINFOnexus (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 13e61863-2ff6-447f-9127-d6350ced40e0 file = nexus/db-queries/src/db/datastore/inventory.rs:1010
50692025-03-02T05:06:01.641ZDEBGnexus (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 13e61863-2ff6-447f-9127-d6350ced40e0 time_started = 2025-03-02 05:06:00.909177 UTC
50702025-03-02T05:06:01.641ZDEBGnexus (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.201378679s iteration = 2 status = {"collection_id":"13e61863-2ff6-447f-9127-d6350ced40e0","time_done":"2025-03-02 05:06:01.447056 UTC","time_started":"2025-03-02 05:06:00.909177 UTC"}
50712025-03-02T05:06:01.641ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 23 reason = Dependency
50722025-03-02T05:06:01.641ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 5 reason = Dependency
50732025-03-02T05:06:01.641ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
50742025-03-02T05:06:01.641ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50752025-03-02T05:06:01.642ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
50762025-03-02T05:06:01.642ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50772025-03-02T05:06:01.648ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50782025-03-02T05:06:01.651ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
50792025-03-02T05:06:01.651ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50802025-03-02T05:06:01.652ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50812025-03-02T05:06:01.660ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50822025-03-02T05:06:01.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50832025-03-02T05:06:01.662ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50842025-03-02T05:06:01.664ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
50852025-03-02T05:06:01.664ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50862025-03-02T05:06:01.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50872025-03-02T05:06:01.697ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50882025-03-02T05:06:01.701ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
50892025-03-02T05:06:01.701ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
50902025-03-02T05:06:01.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
50912025-03-02T05:06:01.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
50922025-03-02T05:06:01.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
50932025-03-02T05:06:01.767ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 126.218084ms iteration = 23 status = {"physical_disks_added":0}
50942025-03-02T05:06:01.790ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
50952025-03-02T05:06:01.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
50962025-03-02T05:06:01.792ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50972025-03-02T05:06:01.794ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
50982025-03-02T05:06:01.794ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
50992025-03-02T05:06:01.794ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51002025-03-02T05:06:01.796ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51012025-03-02T05:06:01.798ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
51022025-03-02T05:06:01.798ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
51032025-03-02T05:06:01.798ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51042025-03-02T05:06:01.800ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
51052025-03-02T05:06:01.801ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51062025-03-02T05:06:01.802ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51072025-03-02T05:06:01.804ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
51082025-03-02T05:06:01.804ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
51092025-03-02T05:06:01.804ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51102025-03-02T05:06:01.805ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51112025-03-02T05:06:01.807ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
51122025-03-02T05:06:01.807ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
51132025-03-02T05:06:01.807ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51142025-03-02T05:06:01.809ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
51152025-03-02T05:06:01.809ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 168.209591ms iteration = 5 status = {"blueprint_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","inventory_collection_id":"af637865-d1e1-4ffd-9583-5c9c32bc0631","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
51162025-03-02T05:06:02.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 264 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = dd87f69f-6fa8-4e08-80af-ec16e9de3a37 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
51172025-03-02T05:06:10.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 62 reason = Timeout
51182025-03-02T05:06:10.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51192025-03-02T05:06:10.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51202025-03-02T05:06:10.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51212025-03-02T05:06:10.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 62 reason = Timeout
51222025-03-02T05:06:10.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
51232025-03-02T05:06:10.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
51242025-03-02T05:06:10.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
51252025-03-02T05:06:10.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51262025-03-02T05:06:10.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51272025-03-02T05:06:10.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51282025-03-02T05:06:10.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51292025-03-02T05:06:10.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51302025-03-02T05:06:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
51312025-03-02T05:06:10.454ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51322025-03-02T05:06:10.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51332025-03-02T05:06:10.469ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
51342025-03-02T05:06:10.469ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 29.16413ms iteration = 62 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
51352025-03-02T05:06:10.481ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
51362025-03-02T05:06:10.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
51372025-03-02T05:06:10.518ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 74.51095ms iteration = 62 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
51382025-03-02T05:06:12.895ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 0bc9e5f1-5fa0-4bbc-9aca-e48b320466a8 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
51392025-03-02T05:06:12.896ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 0bc9e5f1-5fa0-4bbc-9aca-e48b320466a8 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
51402025-03-02T05:06:12.898ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2552 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 0bc9e5f1-5fa0-4bbc-9aca-e48b320466a8 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
51412025-03-02T05:06:12.898ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 8856ab3a-24ae-415c-857b-ab423d7011bd roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
51422025-03-02T05:06:12.898ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 8856ab3a-24ae-415c-857b-ab423d7011bd resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
51432025-03-02T05:06:12.900ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2374 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 8856ab3a-24ae-415c-857b-ab423d7011bd response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
51442025-03-02T05:06:12.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 279 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 3da4cc77-40bd-4d61-8f0a-102ef5801751 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
51452025-03-02T05:06:20.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 63 reason = Timeout
51462025-03-02T05:06:20.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51472025-03-02T05:06:20.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51482025-03-02T05:06:20.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51492025-03-02T05:06:20.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 63 reason = Timeout
51502025-03-02T05:06:20.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
51512025-03-02T05:06:20.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
51522025-03-02T05:06:20.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
51532025-03-02T05:06:20.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51542025-03-02T05:06:20.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51552025-03-02T05:06:20.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51562025-03-02T05:06:20.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51572025-03-02T05:06:20.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51582025-03-02T05:06:20.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
51592025-03-02T05:06:20.450ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51602025-03-02T05:06:20.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51612025-03-02T05:06:20.462ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
51622025-03-02T05:06:20.462ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.974046ms iteration = 63 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
51632025-03-02T05:06:20.475ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
51642025-03-02T05:06:20.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
51652025-03-02T05:06:20.502ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 58.901017ms iteration = 63 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
51662025-03-02T05:06:22.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 453 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 743e0dd0-6d33-4808-bb70-18f3930fd098 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
51672025-03-02T05:06:30.437ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 22 reason = Timeout
51682025-03-02T05:06:30.437ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 22 reason = Timeout
51692025-03-02T05:06:30.437ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 22 reason = Timeout
51702025-03-02T05:06:30.437ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
51712025-03-02T05:06:30.437ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
51722025-03-02T05:06:30.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
51732025-03-02T05:06:30.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
51742025-03-02T05:06:30.439ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3719, tv_nsec: 668275064 } })
51752025-03-02T05:06:30.439ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
51762025-03-02T05:06:30.440ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3719, tv_nsec: 668742081 } })
51772025-03-02T05:06:30.440ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 22 reason = Timeout
51782025-03-02T05:06:30.440ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
51792025-03-02T05:06:30.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 64 reason = Timeout
51802025-03-02T05:06:30.440ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 24 reason = Timeout
51812025-03-02T05:06:30.440ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 22 reason = Timeout
51822025-03-02T05:06:30.440ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
51832025-03-02T05:06:30.440ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
51842025-03-02T05:06:30.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51852025-03-02T05:06:30.440ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
51862025-03-02T05:06:30.440ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.000569ms iteration = 22 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
51872025-03-02T05:06:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
51882025-03-02T05:06:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
51892025-03-02T05:06:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
51902025-03-02T05:06:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51912025-03-02T05:06:30.442ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51922025-03-02T05:06:30.442ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51932025-03-02T05:06:30.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51942025-03-02T05:06:30.442ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51952025-03-02T05:06:30.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 22 reason = Timeout
51962025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
51972025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 22 reason = Timeout
51982025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 22 reason = Timeout
51992025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
52002025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52012025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 22 reason = Timeout
52022025-03-02T05:06:30.444ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
52032025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 22 reason = Timeout
52042025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
52052025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52062025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 22 reason = Timeout
52072025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
52082025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 22 reason = Timeout
52092025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
52102025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 22 reason = Timeout
52112025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
52122025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 22 reason = Timeout
52132025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 22 reason = Timeout
52142025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
52152025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
52162025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
52172025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 64 reason = Timeout
52182025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52192025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 22 reason = Timeout
52202025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
52212025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
52222025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
52232025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
52242025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
52252025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52262025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
52272025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
52282025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52292025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
52302025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
52312025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
52322025-03-02T05:06:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
52332025-03-02T05:06:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
52342025-03-02T05:06:30.445ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
52352025-03-02T05:06:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
52362025-03-02T05:06:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
52372025-03-02T05:06:30.446ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52382025-03-02T05:06:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
52392025-03-02T05:06:30.446ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
52402025-03-02T05:06:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52412025-03-02T05:06:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
52422025-03-02T05:06:30.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
52432025-03-02T05:06:30.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52442025-03-02T05:06:30.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52452025-03-02T05:06:30.450ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52462025-03-02T05:06:30.450ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52472025-03-02T05:06:30.452ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52482025-03-02T05:06:30.453ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
52492025-03-02T05:06:30.453ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52502025-03-02T05:06:30.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52512025-03-02T05:06:30.454ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
52522025-03-02T05:06:30.454ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
52532025-03-02T05:06:30.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
52542025-03-02T05:06:30.455ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
52552025-03-02T05:06:30.455ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52562025-03-02T05:06:30.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52572025-03-02T05:06:30.464ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
52582025-03-02T05:06:30.464ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52592025-03-02T05:06:30.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
52602025-03-02T05:06:30.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52612025-03-02T05:06:30.467ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3719, tv_nsec: 695907214 } })
52622025-03-02T05:06:30.467ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
52632025-03-02T05:06:30.467ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52642025-03-02T05:06:30.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52652025-03-02T05:06:30.506ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 62.294751ms iteration = 22 status = {"errors":[],"garbage_collect_requested":[]}
52662025-03-02T05:06:30.507ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
52672025-03-02T05:06:30.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
52682025-03-02T05:06:30.509ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 65.121268ms iteration = 22 status = {"errors":[],"finish_invoked_ok":[]}
52692025-03-02T05:06:30.513ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52702025-03-02T05:06:30.514ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52712025-03-02T05:06:30.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
52722025-03-02T05:06:30.516ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
52732025-03-02T05:06:30.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
52742025-03-02T05:06:30.518ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
52752025-03-02T05:06:30.518ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52762025-03-02T05:06:30.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52772025-03-02T05:06:30.518ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
52782025-03-02T05:06:30.518ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
52792025-03-02T05:06:30.518ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
52802025-03-02T05:06:30.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
52812025-03-02T05:06:30.520ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 76.379165ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
52822025-03-02T05:06:30.522ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
52832025-03-02T05:06:30.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
52842025-03-02T05:06:30.527ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52852025-03-02T05:06:30.528ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52862025-03-02T05:06:30.528ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52872025-03-02T05:06:30.529ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
52882025-03-02T05:06:30.529ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52892025-03-02T05:06:30.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52902025-03-02T05:06:30.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52912025-03-02T05:06:30.531ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
52922025-03-02T05:06:30.531ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
52932025-03-02T05:06:30.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
52942025-03-02T05:06:30.533ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52952025-03-02T05:06:30.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52962025-03-02T05:06:30.535ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 90.927814ms iteration = 22 status = {"errors":[],"requests_created_ok":[]}
52972025-03-02T05:06:30.537ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
52982025-03-02T05:06:30.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
52992025-03-02T05:06:30.538ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53002025-03-02T05:06:30.540ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
53012025-03-02T05:06:30.540ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
53022025-03-02T05:06:30.541ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a4d01f3-f642-4606-81f8-34f1ec8c2811", "content-length": "26", "date": "Sun, 02 Mar 2025 05:06:30 GMT"} })
53032025-03-02T05:06:30.541ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
53042025-03-02T05:06:30.541ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
53052025-03-02T05:06:30.541ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
53062025-03-02T05:06:30.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
53072025-03-02T05:06:30.543ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
53082025-03-02T05:06:30.543ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 99.0934ms iteration = 22 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
53092025-03-02T05:06:30.543ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
53102025-03-02T05:06:30.543ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
53112025-03-02T05:06:30.543ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
53122025-03-02T05:06:30.543ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 103.117232ms iteration = 64 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
53132025-03-02T05:06:30.543ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
53142025-03-02T05:06:30.543ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "76804997-8bda-4111-952e-ad8094fc8895", "content-length": "26", "date": "Sun, 02 Mar 2025 05:06:30 GMT"} })
53152025-03-02T05:06:30.543ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
53162025-03-02T05:06:30.544ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 104.731014ms iteration = 22 status = 0
53172025-03-02T05:06:30.545ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
53182025-03-02T05:06:30.565ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 121.103147ms iteration = 64 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
53192025-03-02T05:06:30.566ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 122.11082ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
53202025-03-02T05:06:30.571ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
53212025-03-02T05:06:30.571ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
53222025-03-02T05:06:30.571ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbe82a67-657b-4e5b-b680-c730a8b9303d", "content-length": "26", "date": "Sun, 02 Mar 2025 05:06:30 GMT"} })
53232025-03-02T05:06:30.571ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
53242025-03-02T05:06:30.574ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
53252025-03-02T05:06:30.574ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
53262025-03-02T05:06:30.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
53272025-03-02T05:06:30.575ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
53282025-03-02T05:06:30.575ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
53292025-03-02T05:06:30.575ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
53302025-03-02T05:06:30.575ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
53312025-03-02T05:06:30.579ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53322025-03-02T05:06:30.581ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
53332025-03-02T05:06:30.581ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
53342025-03-02T05:06:30.581ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
53352025-03-02T05:06:30.615ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 171.181209ms iteration = 22 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
53362025-03-02T05:06:30.619ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53372025-03-02T05:06:30.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
53382025-03-02T05:06:30.620ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
53392025-03-02T05:06:30.620ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
53402025-03-02T05:06:30.620ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 176.680071ms iteration = 22 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
53412025-03-02T05:06:30.625ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
53422025-03-02T05:06:30.626ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 185.81064ms iteration = 22 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
53432025-03-02T05:06:30.645ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
53442025-03-02T05:06:30.645ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "072fcbd8-a092-44ed-9e31-11b157a0aeea", "content-length": "1", "date": "Sun, 02 Mar 2025 05:06:29 GMT"} })
53452025-03-02T05:06:30.647ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
53462025-03-02T05:06:30.648ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
53472025-03-02T05:06:30.650ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 212.734531ms iteration = 22 status = {}
53482025-03-02T05:06:30.650ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 213.203847ms iteration = 22 status = null
53492025-03-02T05:06:30.652ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 212.427947ms iteration = 24 status = {"physical_disks_added":0}
53502025-03-02T05:06:30.656ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53512025-03-02T05:06:30.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
53522025-03-02T05:06:30.658ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53532025-03-02T05:06:30.660ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
53542025-03-02T05:06:30.660ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53552025-03-02T05:06:30.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
53562025-03-02T05:06:30.662ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53572025-03-02T05:06:30.663ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
53582025-03-02T05:06:30.663ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53592025-03-02T05:06:30.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
53602025-03-02T05:06:30.697ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
53612025-03-02T05:06:30.697ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
53622025-03-02T05:06:30.697ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "d129089a-67af-4e62-a782-3f6b1d99c0c6", "content-length": "372", "date": "Sun, 02 Mar 2025 05:06:30 GMT"} })
53632025-03-02T05:06:30.697ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53642025-03-02T05:06:30.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53652025-03-02T05:06:30.738ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
53662025-03-02T05:06:30.738ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
53672025-03-02T05:06:30.738ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "423812b3-ed06-49c2-8ab8-c4963f1bac99", "date": "Sun, 02 Mar 2025 05:06:30 GMT"} })
53682025-03-02T05:06:30.738ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53692025-03-02T05:06:30.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53702025-03-02T05:06:30.741ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53712025-03-02T05:06:30.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53722025-03-02T05:06:30.744ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53732025-03-02T05:06:30.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53742025-03-02T05:06:30.745ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
53752025-03-02T05:06:30.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
53762025-03-02T05:06:30.746ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
53772025-03-02T05:06:30.746ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
53782025-03-02T05:06:30.747ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "02b2959d-e2e6-4a0c-98c3-aba409d31737", "date": "Sun, 02 Mar 2025 05:06:30 GMT"} })
53792025-03-02T05:06:30.747ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
53802025-03-02T05:06:30.747ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
53812025-03-02T05:06:30.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
53822025-03-02T05:06:30.749ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 305.141524ms iteration = 22 status = {}
53832025-03-02T05:06:30.756ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
53842025-03-02T05:06:30.756ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "659666b7-ed12-49d9-8a69-f626c34bac49", "content-length": "134", "date": "Sun, 02 Mar 2025 05:06:29 GMT"} })
53852025-03-02T05:06:30.756ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
53862025-03-02T05:06:30.756ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
53872025-03-02T05:06:30.757ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
53882025-03-02T05:06:30.757ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
53892025-03-02T05:06:30.757ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
53902025-03-02T05:06:30.757ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
53912025-03-02T05:06:30.757ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
53922025-03-02T05:06:30.757ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
53932025-03-02T05:06:30.757ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
53942025-03-02T05:06:30.757ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
53952025-03-02T05:06:30.757ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
53962025-03-02T05:06:30.757ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
53972025-03-02T05:06:30.757ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
53982025-03-02T05:06:30.759ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
53992025-03-02T05:06:30.759ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "c092104e-cbfc-43bf-9a24-d0a8ab62f018", "content-length": "2", "date": "Sun, 02 Mar 2025 05:06:29 GMT"} })
54002025-03-02T05:06:30.759ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
54012025-03-02T05:06:30.759ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f7b64c8-b208-41a3-a028-5f9a00a45b77", "content-length": "52", "date": "Sun, 02 Mar 2025 05:06:29 GMT"} })
54022025-03-02T05:06:30.759ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
54032025-03-02T05:06:30.759ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
54042025-03-02T05:06:30.783ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
54052025-03-02T05:06:30.852ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "35f161e0-a3b4-445c-a75b-26570e9fd463", "date": "Sun, 02 Mar 2025 05:06:30 GMT"} })
54062025-03-02T05:06:30.852ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
54072025-03-02T05:06:30.852ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
54082025-03-02T05:06:30.852ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "95acfa1f-e60d-482b-a53c-f03667a5b5f0", "content-length": "691", "date": "Sun, 02 Mar 2025 05:06:30 GMT"} })
54092025-03-02T05:06:30.852ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
54102025-03-02T05:06:30.852ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
54112025-03-02T05:06:30.852ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
54122025-03-02T05:06:30.852ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "69b55ce4-c8f0-4076-90a2-cd2d4bd69e48", "content-length": "682", "date": "Sun, 02 Mar 2025 05:06:30 GMT"} })
54132025-03-02T05:06:30.852ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54142025-03-02T05:06:30.853ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54152025-03-02T05:06:30.854ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 410.243824ms iteration = 22 status = {}
54162025-03-02T05:06:30.856ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54172025-03-02T05:06:30.856ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54182025-03-02T05:06:30.859ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54192025-03-02T05:06:30.859ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54202025-03-02T05:06:30.861ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54212025-03-02T05:06:30.861ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54222025-03-02T05:06:30.868ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 424.131655ms iteration = 22 status = {}
54232025-03-02T05:06:32.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 286 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = ac8b8b59-fdb8-45ba-804b-edcdd8115ae7 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
54242025-03-02T05:06:40.441ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 65 reason = Timeout
54252025-03-02T05:06:40.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54262025-03-02T05:06:40.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54272025-03-02T05:06:40.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54282025-03-02T05:06:40.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 65 reason = Timeout
54292025-03-02T05:06:40.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
54302025-03-02T05:06:40.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
54312025-03-02T05:06:40.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
54322025-03-02T05:06:40.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54332025-03-02T05:06:40.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54342025-03-02T05:06:40.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54352025-03-02T05:06:40.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54362025-03-02T05:06:40.450ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54372025-03-02T05:06:40.452ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
54382025-03-02T05:06:40.452ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54392025-03-02T05:06:40.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54402025-03-02T05:06:40.464ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
54412025-03-02T05:06:40.464ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.485257ms iteration = 65 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
54422025-03-02T05:06:40.474ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
54432025-03-02T05:06:40.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
54442025-03-02T05:06:40.476ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.869508ms iteration = 65 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
54452025-03-02T05:06:42.895ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 70929d3c-cf71-476c-bb30-2ea470e06d9f roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
54462025-03-02T05:06:42.896ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 70929d3c-cf71-476c-bb30-2ea470e06d9f resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
54472025-03-02T05:06:42.898ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2987 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 70929d3c-cf71-476c-bb30-2ea470e06d9f response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
54482025-03-02T05:06:42.899ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = ead0d282-5747-4c87-9b5a-5bac4fdd2c9f roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
54492025-03-02T05:06:42.899ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = ead0d282-5747-4c87-9b5a-5bac4fdd2c9f resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
54502025-03-02T05:06:42.901ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2362 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = ead0d282-5747-4c87-9b5a-5bac4fdd2c9f response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
54512025-03-02T05:06:42.902ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 250 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 087906bb-075d-4f47-958e-76d93a36448a response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
54522025-03-02T05:06:50.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 66 reason = Timeout
54532025-03-02T05:06:50.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54542025-03-02T05:06:50.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54552025-03-02T05:06:50.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54562025-03-02T05:06:50.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 66 reason = Timeout
54572025-03-02T05:06:50.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
54582025-03-02T05:06:50.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
54592025-03-02T05:06:50.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
54602025-03-02T05:06:50.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54612025-03-02T05:06:50.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54622025-03-02T05:06:50.445ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
54632025-03-02T05:06:50.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
54642025-03-02T05:06:50.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54652025-03-02T05:06:50.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54662025-03-02T05:06:50.447ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.82119ms iteration = 66 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
54672025-03-02T05:06:50.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54682025-03-02T05:06:50.451ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
54692025-03-02T05:06:50.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54702025-03-02T05:06:50.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54712025-03-02T05:06:50.460ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
54722025-03-02T05:06:50.460ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.424523ms iteration = 66 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
54732025-03-02T05:06:52.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 287 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 0511a486-d0f1-4494-afbd-80661c04cd5c response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
54742025-03-02T05:06:57.896ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 619fe909-9711-480f-b677-ccfeffab5d20 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
54752025-03-02T05:06:57.896ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 619fe909-9711-480f-b677-ccfeffab5d20 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
54762025-03-02T05:06:57.898ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2930 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 619fe909-9711-480f-b677-ccfeffab5d20 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
54772025-03-02T05:06:57.899ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 1af20e35-3437-4537-897a-1743c8519ecd roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
54782025-03-02T05:06:57.899ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 1af20e35-3437-4537-897a-1743c8519ecd resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
54792025-03-02T05:06:57.900ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 1766 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 1af20e35-3437-4537-897a-1743c8519ecd response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
54802025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 23 reason = Timeout
54812025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 12 reason = Timeout
54822025-03-02T05:07:00.438ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-02 04:57:00.438039150 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
54832025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 12 reason = Timeout
54842025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
54852025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
54862025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 23 reason = Timeout
54872025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 14 reason = Timeout
54882025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
54892025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
54902025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 12 reason = Timeout
54912025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 12 reason = Timeout
54922025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
54932025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
54942025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
54952025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 12 reason = Timeout
54962025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 23 reason = Timeout
54972025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
54982025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 14 reason = Timeout
54992025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 12 reason = Timeout
55002025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
55012025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
55022025-03-02T05:07:00.438ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3bf56dce-8353-46bb-a2f0-982fbd876e0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("742da26a-88eb-42ac-9329-e125fe787858.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 3749, tv_nsec: 666950301 } })
55032025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
55042025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
55052025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
55062025-03-02T05:07:00.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
55072025-03-02T05:07:00.438ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4a50e69a-1735-4503-96cb-55c6cc854815.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 3749, tv_nsec: 667108093 } })
55082025-03-02T05:07:00.439ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
55092025-03-02T05:07:00.439ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 892.182\u{b5}s iteration = 12 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
55102025-03-02T05:07:00.439ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
55112025-03-02T05:07:00.439ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 948.263\u{b5}s iteration = 12 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
55122025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55132025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
55142025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
55152025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 25 reason = Timeout
55162025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 67 reason = Timeout
55172025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
55182025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55192025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 12 reason = Timeout
55202025-03-02T05:07:00.440ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
55212025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 13 reason = Timeout
55222025-03-02T05:07:00.440ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca
55232025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
55242025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 23 reason = Timeout
55252025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 9.911\u{b5}s iteration = 13 status = {"enabled":false,"target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca"}
55262025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
55272025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 23 reason = Timeout
55282025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
55292025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
55302025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
55312025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55322025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
55332025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
55342025-03-02T05:07:00.440ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-02 04:57:00.438039150 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
55352025-03-02T05:07:00.440ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 2.885688ms iteration = 12 status = {"expiration":"2025-03-02T04:57:00.438039150Z","pruned":[]}
55362025-03-02T05:07:00.441ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.659257ms iteration = 23 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
55372025-03-02T05:07:00.441ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3749, tv_nsec: 670271714 } })
55382025-03-02T05:07:00.442ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
55392025-03-02T05:07:00.442ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
55402025-03-02T05:07:00.442ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
55412025-03-02T05:07:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
55422025-03-02T05:07:00.442ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3749, tv_nsec: 670947743 } })
55432025-03-02T05:07:00.442ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
55442025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
55452025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
55462025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 23 reason = Timeout
55472025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 23 reason = Timeout
55482025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55492025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
55502025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 23 reason = Timeout
55512025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
55522025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 23 reason = Timeout
55532025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
55542025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 23 reason = Timeout
55552025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
55562025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 23 reason = Timeout
55572025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
55582025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 23 reason = Timeout
55592025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
55602025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 12 reason = Timeout
55612025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 23 reason = Timeout
55622025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
55632025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
55642025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 23 reason = Timeout
55652025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
55662025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
55672025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 67 reason = Timeout
55682025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55692025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 23 reason = Timeout
55702025-03-02T05:07:00.443ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
55712025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
55722025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 12 reason = Timeout
55732025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
55742025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 12 reason = Timeout
55752025-03-02T05:07:00.443ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
55762025-03-02T05:07:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55772025-03-02T05:07:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
55782025-03-02T05:07:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
55792025-03-02T05:07:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
55802025-03-02T05:07:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
55812025-03-02T05:07:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
55822025-03-02T05:07:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
55832025-03-02T05:07:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
55842025-03-02T05:07:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
55852025-03-02T05:07:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
55862025-03-02T05:07:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55872025-03-02T05:07:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
55882025-03-02T05:07:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
55892025-03-02T05:07:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
55902025-03-02T05:07:00.445ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 23 reason = Timeout
55912025-03-02T05:07:00.445ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
55922025-03-02T05:07:00.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
55932025-03-02T05:07:00.451ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
55942025-03-02T05:07:00.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
55952025-03-02T05:07:00.452ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55962025-03-02T05:07:00.452ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55972025-03-02T05:07:00.453ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55982025-03-02T05:07:00.453ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55992025-03-02T05:07:00.453ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56002025-03-02T05:07:00.453ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56012025-03-02T05:07:00.453ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56022025-03-02T05:07:00.453ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56032025-03-02T05:07:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
56042025-03-02T05:07:00.457ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56052025-03-02T05:07:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56062025-03-02T05:07:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
56072025-03-02T05:07:00.457ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56082025-03-02T05:07:00.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56092025-03-02T05:07:00.458ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
56102025-03-02T05:07:00.458ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
56112025-03-02T05:07:00.458ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
56122025-03-02T05:07:00.458ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
56132025-03-02T05:07:00.458ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
56142025-03-02T05:07:00.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56152025-03-02T05:07:00.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
56162025-03-02T05:07:00.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
56172025-03-02T05:07:00.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56182025-03-02T05:07:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
56192025-03-02T05:07:00.459ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56202025-03-02T05:07:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
56212025-03-02T05:07:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
56222025-03-02T05:07:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56232025-03-02T05:07:00.461ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
56242025-03-02T05:07:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
56252025-03-02T05:07:00.462ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56262025-03-02T05:07:00.465ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
56272025-03-02T05:07:00.465ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56282025-03-02T05:07:00.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56292025-03-02T05:07:00.466ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 22.557254ms iteration = 12 status = {"errors":[],"found_port_ok":[]}
56302025-03-02T05:07:00.473ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56312025-03-02T05:07:00.474ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3749, tv_nsec: 702672836 } })
56322025-03-02T05:07:00.474ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
56332025-03-02T05:07:00.475ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56342025-03-02T05:07:00.475ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
56352025-03-02T05:07:00.475ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
56362025-03-02T05:07:00.475ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 31.797944ms iteration = 12 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
56372025-03-02T05:07:00.475ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56382025-03-02T05:07:00.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56392025-03-02T05:07:00.476ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
56402025-03-02T05:07:00.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56412025-03-02T05:07:00.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
56422025-03-02T05:07:00.476ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56432025-03-02T05:07:00.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56442025-03-02T05:07:00.478ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
56452025-03-02T05:07:00.478ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
56462025-03-02T05:07:00.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
56472025-03-02T05:07:00.483ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56482025-03-02T05:07:00.486ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
56492025-03-02T05:07:00.486ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56502025-03-02T05:07:00.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56512025-03-02T05:07:00.487ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56522025-03-02T05:07:00.490ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
56532025-03-02T05:07:00.490ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56542025-03-02T05:07:00.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56552025-03-02T05:07:00.505ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56562025-03-02T05:07:00.507ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
56572025-03-02T05:07:00.507ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56582025-03-02T05:07:00.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56592025-03-02T05:07:00.512ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 69.021239ms iteration = 23 status = {"errors":[],"garbage_collect_requested":[]}
56602025-03-02T05:07:00.514ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56612025-03-02T05:07:00.517ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
56622025-03-02T05:07:00.517ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
56632025-03-02T05:07:00.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
56642025-03-02T05:07:00.523ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 80.007092ms iteration = 23 status = {"errors":[],"finish_invoked_ok":[]}
56652025-03-02T05:07:00.527ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
56662025-03-02T05:07:00.527ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
56672025-03-02T05:07:00.527ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
56682025-03-02T05:07:00.527ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
56692025-03-02T05:07:00.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
56702025-03-02T05:07:00.528ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
56712025-03-02T05:07:00.528ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
56722025-03-02T05:07:00.528ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
56732025-03-02T05:07:00.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
56742025-03-02T05:07:00.536ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
56752025-03-02T05:07:00.536ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 95.846159ms iteration = 67 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
56762025-03-02T05:07:00.538ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56772025-03-02T05:07:00.538ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
56782025-03-02T05:07:00.538ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
56792025-03-02T05:07:00.538ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
56802025-03-02T05:07:00.538ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
56812025-03-02T05:07:00.538ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
56822025-03-02T05:07:00.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56832025-03-02T05:07:00.538ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56842025-03-02T05:07:00.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
56852025-03-02T05:07:00.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
56862025-03-02T05:07:00.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56872025-03-02T05:07:00.539ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
56882025-03-02T05:07:00.539ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
56892025-03-02T05:07:00.539ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 99.471226ms iteration = 12 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
56902025-03-02T05:07:00.543ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 98.420832ms iteration = 23 status = {"errors":[],"requests_created_ok":[]}
56912025-03-02T05:07:00.544ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
56922025-03-02T05:07:00.544ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
56932025-03-02T05:07:00.546ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
56942025-03-02T05:07:00.546ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
56952025-03-02T05:07:00.562ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
56962025-03-02T05:07:00.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
56972025-03-02T05:07:00.564ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 120.337417ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
56982025-03-02T05:07:00.595ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
56992025-03-02T05:07:00.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
57002025-03-02T05:07:00.617ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57012025-03-02T05:07:00.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57022025-03-02T05:07:00.634ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
57032025-03-02T05:07:00.650ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
57042025-03-02T05:07:00.650ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
57052025-03-02T05:07:00.650ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
57062025-03-02T05:07:00.650ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
57072025-03-02T05:07:00.651ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7447597-68ca-4778-9d0c-34498b983180", "content-length": "26", "date": "Sun, 02 Mar 2025 05:07:00 GMT"} })
57082025-03-02T05:07:00.651ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
57092025-03-02T05:07:00.651ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "322db5d2-c98c-41ed-b15b-42a62c716c2b", "content-length": "26", "date": "Sun, 02 Mar 2025 05:07:00 GMT"} })
57102025-03-02T05:07:00.651ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
57112025-03-02T05:07:00.652ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
57122025-03-02T05:07:00.653ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
57132025-03-02T05:07:00.680ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
57142025-03-02T05:07:00.680ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
57152025-03-02T05:07:00.681ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "a803951d-7d4c-4368-a5be-f4357f35f1ce", "content-length": "372", "date": "Sun, 02 Mar 2025 05:07:00 GMT"} })
57162025-03-02T05:07:00.681ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57172025-03-02T05:07:00.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57182025-03-02T05:07:00.683ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57192025-03-02T05:07:00.685ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57202025-03-02T05:07:00.685ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57212025-03-02T05:07:00.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57222025-03-02T05:07:00.686ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
57232025-03-02T05:07:00.686ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
57242025-03-02T05:07:00.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
57252025-03-02T05:07:00.687ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
57262025-03-02T05:07:00.687ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
57272025-03-02T05:07:00.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
57282025-03-02T05:07:00.718ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
57292025-03-02T05:07:00.719ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
57302025-03-02T05:07:00.719ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "88b4abcf-e7d5-4de3-9a7a-0ebdef2e9821", "date": "Sun, 02 Mar 2025 05:07:00 GMT"} })
57312025-03-02T05:07:00.719ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57322025-03-02T05:07:00.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57332025-03-02T05:07:00.722ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57342025-03-02T05:07:00.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57352025-03-02T05:07:00.726ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57362025-03-02T05:07:00.727ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57372025-03-02T05:07:00.731ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
57382025-03-02T05:07:00.731ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
57392025-03-02T05:07:00.732ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "2b5123df-1f18-43cf-9fa0-b601545bd42a", "date": "Sun, 02 Mar 2025 05:07:00 GMT"} })
57402025-03-02T05:07:00.732ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
57412025-03-02T05:07:00.732ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
57422025-03-02T05:07:00.732ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
57432025-03-02T05:07:00.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
57442025-03-02T05:07:00.733ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 289.650603ms iteration = 23 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
57452025-03-02T05:07:00.734ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 290.408063ms iteration = 23 status = {}
57462025-03-02T05:07:00.737ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
57472025-03-02T05:07:00.737ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
57482025-03-02T05:07:00.737ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
57492025-03-02T05:07:00.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
57502025-03-02T05:07:00.737ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57512025-03-02T05:07:00.738ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57522025-03-02T05:07:00.740ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
57532025-03-02T05:07:00.740ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
57542025-03-02T05:07:00.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
57552025-03-02T05:07:00.741ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 298.074533ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
57562025-03-02T05:07:00.742ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
57572025-03-02T05:07:00.742ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 298.802432ms iteration = 23 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
57582025-03-02T05:07:00.742ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
57592025-03-02T05:07:00.742ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
57602025-03-02T05:07:00.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
57612025-03-02T05:07:00.742ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57622025-03-02T05:07:00.743ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
57632025-03-02T05:07:00.743ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 302.763173ms iteration = 23 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
57642025-03-02T05:07:00.744ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
57652025-03-02T05:07:00.744ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
57662025-03-02T05:07:00.744ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b68f550-8615-4aaf-a039-9e854ad83cdc", "content-length": "26", "date": "Sun, 02 Mar 2025 05:07:00 GMT"} })
57672025-03-02T05:07:00.744ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
57682025-03-02T05:07:00.745ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
57692025-03-02T05:07:00.745ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57702025-03-02T05:07:00.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57712025-03-02T05:07:00.746ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 302.892336ms iteration = 67 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
57722025-03-02T05:07:00.747ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
57732025-03-02T05:07:00.747ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 306.935338ms iteration = 23 status = 0
57742025-03-02T05:07:00.749ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
57752025-03-02T05:07:00.764ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57762025-03-02T05:07:00.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
57772025-03-02T05:07:00.768ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
57782025-03-02T05:07:00.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
57792025-03-02T05:07:00.773ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
57802025-03-02T05:07:00.773ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
57812025-03-02T05:07:00.773ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-02 04:56:25.487890 UTC dns_group = internal generation = 1
57822025-03-02T05:07:00.773ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 335.175276ms iteration = 12 status = {"generation":1}
57832025-03-02T05:07:00.783ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
57842025-03-02T05:07:00.783ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 339.298699ms iteration = 12 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
57852025-03-02T05:07:00.790ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
57862025-03-02T05:07:00.790ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
57872025-03-02T05:07:00.790ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 346.22926ms iteration = 23 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
57882025-03-02T05:07:00.791ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-02T04:55:57.280988Z, time_modified: 2025-03-02T04:55:57.280988Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:94
57892025-03-02T05:07:00.791ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 353.536465ms iteration = 12 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]}},"default_endpoint":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
57902025-03-02T05:07:00.793ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
57912025-03-02T05:07:00.793ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 0 version = 2
57922025-03-02T05:07:00.793ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
57932025-03-02T05:07:00.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
57942025-03-02T05:07:00.795ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57952025-03-02T05:07:00.799ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
57962025-03-02T05:07:00.799ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
57972025-03-02T05:07:00.799ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
57982025-03-02T05:07:00.801ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 1 version = 2
57992025-03-02T05:07:00.801ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
58002025-03-02T05:07:00.801ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-02 04:56:26.100923 UTC dns_group = external generation = 2
58012025-03-02T05:07:00.801ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 362.917657ms iteration = 12 status = {"generation":2}
58022025-03-02T05:07:00.819ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 381.784043ms iteration = 23 status = {}
58032025-03-02T05:07:00.830ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 390.01017ms iteration = 25 status = {"physical_disks_added":0}
58042025-03-02T05:07:00.833ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
58052025-03-02T05:07:00.833ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
58062025-03-02T05:07:00.834ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58072025-03-02T05:07:00.836ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
58082025-03-02T05:07:00.836ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
58092025-03-02T05:07:00.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
58102025-03-02T05:07:00.867ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58112025-03-02T05:07:00.868ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
58122025-03-02T05:07:00.869ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
58132025-03-02T05:07:00.869ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
58142025-03-02T05:07:00.903ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
58152025-03-02T05:07:00.903ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4a60591-69f2-4023-ac0b-6bcf95d4f2b2", "content-length": "1", "date": "Sun, 02 Mar 2025 05:06:59 GMT"} })
58162025-03-02T05:07:00.905ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
58172025-03-02T05:07:00.905ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
58182025-03-02T05:07:00.907ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 469.706949ms iteration = 23 status = null
58192025-03-02T05:07:00.907ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
58202025-03-02T05:07:00.908ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "33b8828c-fe2f-408d-96e3-13883b1a594a", "date": "Sun, 02 Mar 2025 05:07:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
58212025-03-02T05:07:00.908ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
58222025-03-02T05:07:00.909ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "f43fdb46-2042-4b4d-b00a-f68b61a9ce7f", "date": "Sun, 02 Mar 2025 05:07:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
58232025-03-02T05:07:00.909ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
58242025-03-02T05:07:00.910ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
58252025-03-02T05:07:00.910ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 472.91148ms iteration = 14 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
58262025-03-02T05:07:00.926ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
58272025-03-02T05:07:00.979ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
58282025-03-02T05:07:00.980ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d1b7565-f7ef-4972-8e35-6f74527c61b2", "content-length": "691", "date": "Sun, 02 Mar 2025 05:07:00 GMT"} })
58292025-03-02T05:07:00.980ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
58302025-03-02T05:07:00.980ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
58312025-03-02T05:07:00.980ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
58322025-03-02T05:07:00.981ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 538.199421ms iteration = 23 status = {}
58332025-03-02T05:07:01.072ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
58342025-03-02T05:07:01.072ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "1d84d806-4852-415c-8178-0a6aa9b5a352", "date": "Sun, 02 Mar 2025 05:07:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
58352025-03-02T05:07:01.073ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
58362025-03-02T05:07:01.075ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "45938747-8e6b-4de2-a66b-8985e8a0ef34", "date": "Sun, 02 Mar 2025 05:07:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
58372025-03-02T05:07:01.075ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
58382025-03-02T05:07:01.076ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "7249985c-cdcd-4f9a-9ae8-7cc65036cb44", "date": "Sun, 02 Mar 2025 05:07:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
58392025-03-02T05:07:01.076ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
58402025-03-02T05:07:01.078ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
58412025-03-02T05:07:01.078ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 640.285161ms iteration = 14 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
58422025-03-02T05:07:01.107ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58432025-03-02T05:07:01.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58442025-03-02T05:07:01.109ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
58452025-03-02T05:07:01.109ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58462025-03-02T05:07:01.110ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58472025-03-02T05:07:01.119ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
58482025-03-02T05:07:01.120ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "37277510-c301-4ba8-8add-09b6d6bd25eb", "content-length": "134", "date": "Sun, 02 Mar 2025 05:07:00 GMT"} })
58492025-03-02T05:07:01.120ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
58502025-03-02T05:07:01.120ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
58512025-03-02T05:07:01.120ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
58522025-03-02T05:07:01.120ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
58532025-03-02T05:07:01.120ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58542025-03-02T05:07:01.120ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58552025-03-02T05:07:01.120ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58562025-03-02T05:07:01.120ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
58572025-03-02T05:07:01.120ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58582025-03-02T05:07:01.120ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
58592025-03-02T05:07:01.121ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
58602025-03-02T05:07:01.121ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58612025-03-02T05:07:01.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58622025-03-02T05:07:01.122ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
58632025-03-02T05:07:01.122ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb2dfdff-df87-47b8-a411-348e1297702b", "content-length": "2", "date": "Sun, 02 Mar 2025 05:07:00 GMT"} })
58642025-03-02T05:07:01.122ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
58652025-03-02T05:07:01.123ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "079dfcf4-07b6-4117-adde-c53afef88b9f", "content-length": "52", "date": "Sun, 02 Mar 2025 05:07:01 GMT"} })
58662025-03-02T05:07:01.123ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
58672025-03-02T05:07:01.123ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
58682025-03-02T05:07:01.259ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
58692025-03-02T05:07:01.259ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
58702025-03-02T05:07:01.290ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
58712025-03-02T05:07:01.290ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:35981
58722025-03-02T05:07:01.290ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "15bd06d0-1b0a-4998-b8c3-998b89faaefe", "date": "Sun, 02 Mar 2025 05:07:00 GMT"} })
58732025-03-02T05:07:01.290ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
58742025-03-02T05:07:01.290ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
58752025-03-02T05:07:01.290ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:35981 req_id = 27c14e98-8962-4f48-844e-c4f03b4a88dc roles = RoleSet { roles: {} } uri = /metrics/producers
58762025-03-02T05:07:01.290ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "906007a2-9822-41fc-8908-36c8819ebf4a", "content-length": "682", "date": "Sun, 02 Mar 2025 05:07:01 GMT"} })
58772025-03-02T05:07:01.290ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58782025-03-02T05:07:01.291ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:35981 req_id = 27c14e98-8962-4f48-844e-c4f03b4a88dc resource = Database result = Ok(()) uri = /metrics/producers
58792025-03-02T05:07:01.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58802025-03-02T05:07:01.294ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58812025-03-02T05:07:01.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58822025-03-02T05:07:01.297ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58832025-03-02T05:07:01.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58842025-03-02T05:07:01.299ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58852025-03-02T05:07:01.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58862025-03-02T05:07:01.306ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 862.597867ms iteration = 23 status = {}
58872025-03-02T05:07:01.327ZINFOnexus: assigned collector to new producer collector_id = 8aaa536c-c534-4dd2-aad0-92d8934e5eba file = nexus/src/app/oximeter.rs:76 producer_id = aebed526-5e03-4a58-86b5-1baf17c88392
58882025-03-02T05:07:01.327ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 36707 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:35981 req_id = 27c14e98-8962-4f48-844e-c4f03b4a88dc response_code = 201 uri = /metrics/producers
58892025-03-02T05:07:01.327ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "27c14e98-8962-4f48-844e-c4f03b4a88dc", "content-length": "41", "date": "Sun, 02 Mar 2025 05:07:01 GMT"} })
58902025-03-02T05:07:01.328ZDEBGnexus (producer-registration-task): registered with nexus successfully
58912025-03-02T05:07:01.328ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
58922025-03-02T05:07:02.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 264 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 048c264c-cf88-42c1-ab0d-0250bb67660a response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
58932025-03-02T05:07:10.441ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 68 reason = Timeout
58942025-03-02T05:07:10.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58952025-03-02T05:07:10.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58962025-03-02T05:07:10.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58972025-03-02T05:07:10.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 68 reason = Timeout
58982025-03-02T05:07:10.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58992025-03-02T05:07:10.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59002025-03-02T05:07:10.445ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59012025-03-02T05:07:10.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
59022025-03-02T05:07:10.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59032025-03-02T05:07:10.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59042025-03-02T05:07:10.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59052025-03-02T05:07:10.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59062025-03-02T05:07:10.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59072025-03-02T05:07:10.450ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59082025-03-02T05:07:10.452ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
59092025-03-02T05:07:10.452ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59102025-03-02T05:07:10.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59112025-03-02T05:07:10.465ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
59122025-03-02T05:07:10.465ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.460098ms iteration = 68 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
59132025-03-02T05:07:10.478ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 35.396711ms iteration = 68 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
59142025-03-02T05:07:12.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 261 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 8bd91d54-e9ff-4e10-9974-781b36687584 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
59152025-03-02T05:07:20.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 69 reason = Timeout
59162025-03-02T05:07:20.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59172025-03-02T05:07:20.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59182025-03-02T05:07:20.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 69 reason = Timeout
59192025-03-02T05:07:20.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59202025-03-02T05:07:20.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59212025-03-02T05:07:20.445ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59222025-03-02T05:07:20.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59232025-03-02T05:07:20.447ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.554906ms iteration = 69 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
59242025-03-02T05:07:20.470ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59252025-03-02T05:07:20.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
59262025-03-02T05:07:20.473ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59272025-03-02T05:07:20.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59282025-03-02T05:07:20.475ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59292025-03-02T05:07:20.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59302025-03-02T05:07:20.476ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59312025-03-02T05:07:20.479ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
59322025-03-02T05:07:20.479ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59332025-03-02T05:07:20.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59342025-03-02T05:07:20.486ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
59352025-03-02T05:07:20.486ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 45.756827ms iteration = 69 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
59362025-03-02T05:07:22.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 290 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 45c3983c-03f0-4ac6-8407-5ca75905b574 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
59372025-03-02T05:07:27.896ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = b6330f76-748d-407b-a32a-c07235b83238 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
59382025-03-02T05:07:27.897ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = b6330f76-748d-407b-a32a-c07235b83238 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
59392025-03-02T05:07:27.900ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3838 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = b6330f76-748d-407b-a32a-c07235b83238 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
59402025-03-02T05:07:27.900ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 8467af59-a146-4681-a1fd-4001d8456663 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
59412025-03-02T05:07:27.901ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 8467af59-a146-4681-a1fd-4001d8456663 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
59422025-03-02T05:07:27.902ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2230 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61257 req_id = 8467af59-a146-4681-a1fd-4001d8456663 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
59432025-03-02T05:07:30.437ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 24 reason = Timeout
59442025-03-02T05:07:30.437ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 24 reason = Timeout
59452025-03-02T05:07:30.437ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
59462025-03-02T05:07:30.437ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 24 reason = Timeout
59472025-03-02T05:07:30.437ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
59482025-03-02T05:07:30.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
59492025-03-02T05:07:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
59502025-03-02T05:07:30.440ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 26 reason = Timeout
59512025-03-02T05:07:30.440ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3779, tv_nsec: 668759606 } })
59522025-03-02T05:07:30.440ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 70 reason = Timeout
59532025-03-02T05:07:30.440ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 24 reason = Timeout
59542025-03-02T05:07:30.440ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
59552025-03-02T05:07:30.440ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 24 reason = Timeout
59562025-03-02T05:07:30.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59572025-03-02T05:07:30.440ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
59582025-03-02T05:07:30.440ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
59592025-03-02T05:07:30.440ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3779, tv_nsec: 668949048 } })
59602025-03-02T05:07:30.440ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
59612025-03-02T05:07:30.440ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.221432ms iteration = 24 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
59622025-03-02T05:07:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
59632025-03-02T05:07:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
59642025-03-02T05:07:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59652025-03-02T05:07:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
59662025-03-02T05:07:30.441ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
59672025-03-02T05:07:30.442ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59682025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59692025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 24 reason = Timeout
59702025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
59712025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 24 reason = Timeout
59722025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59732025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
59742025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 24 reason = Timeout
59752025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
59762025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 24 reason = Timeout
59772025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
59782025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
59792025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 70 reason = Timeout
59802025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59812025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 24 reason = Timeout
59822025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
59832025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 24 reason = Timeout
59842025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
59852025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 24 reason = Timeout
59862025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
59872025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 24 reason = Timeout
59882025-03-02T05:07:30.443ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
59892025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 24 reason = Timeout
59902025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59912025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
59922025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 24 reason = Timeout
59932025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
59942025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
59952025-03-02T05:07:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
59962025-03-02T05:07:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
59972025-03-02T05:07:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59982025-03-02T05:07:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
59992025-03-02T05:07:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
60002025-03-02T05:07:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
60012025-03-02T05:07:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
60022025-03-02T05:07:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
60032025-03-02T05:07:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
60042025-03-02T05:07:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60052025-03-02T05:07:30.444ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 24 reason = Timeout
60062025-03-02T05:07:30.444ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
60072025-03-02T05:07:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
60082025-03-02T05:07:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
60092025-03-02T05:07:30.445ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
60102025-03-02T05:07:30.445ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60112025-03-02T05:07:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60122025-03-02T05:07:30.446ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60132025-03-02T05:07:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
60142025-03-02T05:07:30.446ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
60152025-03-02T05:07:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
60162025-03-02T05:07:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
60172025-03-02T05:07:30.448ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
60182025-03-02T05:07:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
60192025-03-02T05:07:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
60202025-03-02T05:07:30.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60212025-03-02T05:07:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
60222025-03-02T05:07:30.448ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
60232025-03-02T05:07:30.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60242025-03-02T05:07:30.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
60252025-03-02T05:07:30.452ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
60262025-03-02T05:07:30.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
60272025-03-02T05:07:30.454ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60282025-03-02T05:07:30.457ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
60292025-03-02T05:07:30.457ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
60302025-03-02T05:07:30.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60312025-03-02T05:07:30.458ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3779, tv_nsec: 686709800 } })
60322025-03-02T05:07:30.458ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
60332025-03-02T05:07:30.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60342025-03-02T05:07:30.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60352025-03-02T05:07:30.463ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
60362025-03-02T05:07:30.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
60372025-03-02T05:07:30.465ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60382025-03-02T05:07:30.468ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
60392025-03-02T05:07:30.468ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
60402025-03-02T05:07:30.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60412025-03-02T05:07:30.469ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
60422025-03-02T05:07:30.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60432025-03-02T05:07:30.470ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60442025-03-02T05:07:30.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
60452025-03-02T05:07:30.473ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60462025-03-02T05:07:30.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60472025-03-02T05:07:30.474ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60482025-03-02T05:07:30.476ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
60492025-03-02T05:07:30.476ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
60502025-03-02T05:07:30.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60512025-03-02T05:07:30.490ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
60522025-03-02T05:07:30.490ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 50.372356ms iteration = 70 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
60532025-03-02T05:07:30.496ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
60542025-03-02T05:07:30.496ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
60552025-03-02T05:07:30.496ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
60562025-03-02T05:07:30.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
60572025-03-02T05:07:30.506ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60582025-03-02T05:07:30.508ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
60592025-03-02T05:07:30.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
60602025-03-02T05:07:30.509ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
60612025-03-02T05:07:30.509ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60622025-03-02T05:07:30.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60632025-03-02T05:07:30.511ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 67.921905ms iteration = 24 status = {"errors":[],"finish_invoked_ok":[]}
60642025-03-02T05:07:30.515ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60652025-03-02T05:07:30.515ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 71.838786ms iteration = 24 status = {"errors":[],"garbage_collect_requested":[]}
60662025-03-02T05:07:30.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60672025-03-02T05:07:30.519ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
60682025-03-02T05:07:30.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
60692025-03-02T05:07:30.520ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
60702025-03-02T05:07:30.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
60712025-03-02T05:07:30.522ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60722025-03-02T05:07:30.523ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
60732025-03-02T05:07:30.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
60742025-03-02T05:07:30.524ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
60752025-03-02T05:07:30.524ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
60762025-03-02T05:07:30.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
60772025-03-02T05:07:30.527ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 83.048772ms iteration = 24 status = {"errors":[],"requests_created_ok":[]}
60782025-03-02T05:07:30.529ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
60792025-03-02T05:07:30.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
60802025-03-02T05:07:30.530ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
60812025-03-02T05:07:30.530ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 86.938202ms iteration = 24 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
60822025-03-02T05:07:30.537ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
60832025-03-02T05:07:30.537ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
60842025-03-02T05:07:30.538ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f3b8f91-56a9-4e72-8ca0-3320081c570d", "content-length": "26", "date": "Sun, 02 Mar 2025 05:07:30 GMT"} })
60852025-03-02T05:07:30.538ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
60862025-03-02T05:07:30.540ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
60872025-03-02T05:07:30.557ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60882025-03-02T05:07:30.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60892025-03-02T05:07:30.558ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
60902025-03-02T05:07:30.558ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
60912025-03-02T05:07:30.558ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
60922025-03-02T05:07:30.558ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
60932025-03-02T05:07:30.558ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
60942025-03-02T05:07:30.558ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
60952025-03-02T05:07:30.558ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "32f69d1a-db26-49a5-8eb7-7736bac082fc", "content-length": "26", "date": "Sun, 02 Mar 2025 05:07:30 GMT"} })
60962025-03-02T05:07:30.558ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
60972025-03-02T05:07:30.560ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
60982025-03-02T05:07:30.560ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60992025-03-02T05:07:30.560ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
61002025-03-02T05:07:30.562ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61012025-03-02T05:07:30.563ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 119.823471ms iteration = 70 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
61022025-03-02T05:07:30.564ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 121.170478ms iteration = 24 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
61032025-03-02T05:07:30.565ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
61042025-03-02T05:07:30.565ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
61052025-03-02T05:07:30.565ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
61062025-03-02T05:07:30.567ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 126.502268ms iteration = 24 status = 0
61072025-03-02T05:07:30.567ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 123.656611ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
61082025-03-02T05:07:30.573ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
61092025-03-02T05:07:30.573ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 133.342487ms iteration = 24 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
61102025-03-02T05:07:30.583ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 140.045904ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
61112025-03-02T05:07:30.590ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
61122025-03-02T05:07:30.590ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
61132025-03-02T05:07:30.590ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "754ece46-138a-44d0-93f9-6bca48986c5a", "content-length": "26", "date": "Sun, 02 Mar 2025 05:07:30 GMT"} })
61142025-03-02T05:07:30.590ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
61152025-03-02T05:07:30.590ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
61162025-03-02T05:07:30.590ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
61172025-03-02T05:07:30.590ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 147.086656ms iteration = 24 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
61182025-03-02T05:07:30.593ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
61192025-03-02T05:07:30.607ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 167.176757ms iteration = 26 status = {"physical_disks_added":0}
61202025-03-02T05:07:30.609ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
61212025-03-02T05:07:30.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
61222025-03-02T05:07:30.611ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61232025-03-02T05:07:30.613ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
61242025-03-02T05:07:30.613ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
61252025-03-02T05:07:30.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
61262025-03-02T05:07:30.614ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61272025-03-02T05:07:30.616ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
61282025-03-02T05:07:30.616ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
61292025-03-02T05:07:30.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
61302025-03-02T05:07:30.652ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 215.343485ms iteration = 24 status = {}
61312025-03-02T05:07:30.675ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
61322025-03-02T05:07:30.675ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
61332025-03-02T05:07:30.675ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "59dfe9de-e79b-440d-a604-b0df02ddd895", "content-length": "372", "date": "Sun, 02 Mar 2025 05:07:30 GMT"} })
61342025-03-02T05:07:30.675ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61352025-03-02T05:07:30.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61362025-03-02T05:07:30.684ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
61372025-03-02T05:07:30.684ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "eeee0b88-7dc8-45dc-8237-132b5dd907dc", "content-length": "2", "date": "Sun, 02 Mar 2025 05:07:30 GMT"} })
61382025-03-02T05:07:30.686ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
61392025-03-02T05:07:30.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
61402025-03-02T05:07:30.718ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
61412025-03-02T05:07:30.719ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "29ac8b3c-bfa0-4603-b40b-1017b3db5b43", "content-length": "691", "date": "Sun, 02 Mar 2025 05:07:30 GMT"} })
61422025-03-02T05:07:30.719ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
61432025-03-02T05:07:30.719ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
61442025-03-02T05:07:30.719ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
61452025-03-02T05:07:30.721ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 277.410084ms iteration = 24 status = {}
61462025-03-02T05:07:30.721ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 283.630375ms iteration = 24 status = null
61472025-03-02T05:07:30.725ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
61482025-03-02T05:07:30.725ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
61492025-03-02T05:07:30.725ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "1888c2e8-230d-4eb8-8eaa-0ab4d798b9b5", "date": "Sun, 02 Mar 2025 05:07:30 GMT"} })
61502025-03-02T05:07:30.725ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61512025-03-02T05:07:30.726ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61522025-03-02T05:07:30.728ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61532025-03-02T05:07:30.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61542025-03-02T05:07:30.731ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61552025-03-02T05:07:30.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61562025-03-02T05:07:30.733ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
61572025-03-02T05:07:30.733ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
61582025-03-02T05:07:30.734ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "7c1c2318-be4f-4f4d-b555-cda9180cecf2", "date": "Sun, 02 Mar 2025 05:07:30 GMT"} })
61592025-03-02T05:07:30.736ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 292.890445ms iteration = 24 status = {}
61602025-03-02T05:07:30.760ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61612025-03-02T05:07:30.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61622025-03-02T05:07:30.763ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
61632025-03-02T05:07:30.763ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61642025-03-02T05:07:30.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61652025-03-02T05:07:30.773ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
61662025-03-02T05:07:30.773ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f71d63a-0890-41fe-a334-804825e0d03e", "content-length": "134", "date": "Sun, 02 Mar 2025 05:07:30 GMT"} })
61672025-03-02T05:07:30.773ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
61682025-03-02T05:07:30.773ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
61692025-03-02T05:07:30.774ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
61702025-03-02T05:07:30.774ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
61712025-03-02T05:07:30.774ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
61722025-03-02T05:07:30.774ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
61732025-03-02T05:07:30.774ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
61742025-03-02T05:07:30.774ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
61752025-03-02T05:07:30.774ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
61762025-03-02T05:07:30.774ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
61772025-03-02T05:07:30.774ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
61782025-03-02T05:07:30.774ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61792025-03-02T05:07:30.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61802025-03-02T05:07:30.776ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
61812025-03-02T05:07:30.777ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "edced0f8-42c0-444a-abbf-08a3386150a7", "content-length": "2", "date": "Sun, 02 Mar 2025 05:07:30 GMT"} })
61822025-03-02T05:07:30.777ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
61832025-03-02T05:07:30.777ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc1728bd-2126-4eb4-b9a2-aa80b5efd9b9", "content-length": "52", "date": "Sun, 02 Mar 2025 05:07:30 GMT"} })
61842025-03-02T05:07:30.777ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
61852025-03-02T05:07:30.777ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
61862025-03-02T05:07:30.871ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "8f6af1d3-1d78-4610-b1b3-60b365cc2f93", "date": "Sun, 02 Mar 2025 05:07:30 GMT"} })
61872025-03-02T05:07:30.871ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
61882025-03-02T05:07:30.871ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
61892025-03-02T05:07:30.876ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5c7301c-5617-4e96-afc8-748b3d02f480", "content-length": "682", "date": "Sun, 02 Mar 2025 05:07:30 GMT"} })
61902025-03-02T05:07:30.876ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61912025-03-02T05:07:30.876ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61922025-03-02T05:07:30.879ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61932025-03-02T05:07:30.880ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61942025-03-02T05:07:30.883ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61952025-03-02T05:07:30.883ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61962025-03-02T05:07:30.885ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61972025-03-02T05:07:30.886ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61982025-03-02T05:07:30.892ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 449.13577ms iteration = 24 status = {}
61992025-03-02T05:07:32.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 307 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 771d6702-d417-4d72-9eed-1ede0571ba06 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
62002025-03-02T05:07:40.441ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 71 reason = Timeout
62012025-03-02T05:07:40.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62022025-03-02T05:07:40.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62032025-03-02T05:07:40.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 71 reason = Timeout
62042025-03-02T05:07:40.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62052025-03-02T05:07:40.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62062025-03-02T05:07:40.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62072025-03-02T05:07:40.447ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62082025-03-02T05:07:40.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62092025-03-02T05:07:40.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
62102025-03-02T05:07:40.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62112025-03-02T05:07:40.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62122025-03-02T05:07:40.450ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 6.077609ms iteration = 71 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
62132025-03-02T05:07:40.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62142025-03-02T05:07:40.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62152025-03-02T05:07:40.453ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62162025-03-02T05:07:40.455ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
62172025-03-02T05:07:40.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62182025-03-02T05:07:40.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62192025-03-02T05:07:40.464ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
62202025-03-02T05:07:40.464ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.139812ms iteration = 71 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
62212025-03-02T05:07:42.904ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 317 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = d1241182-6d6e-4e9c-9b04-f960d51ef792 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
62222025-03-02T05:07:50.441ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 72 reason = Timeout
62232025-03-02T05:07:50.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62242025-03-02T05:07:50.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62252025-03-02T05:07:50.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62262025-03-02T05:07:50.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 72 reason = Timeout
62272025-03-02T05:07:50.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62282025-03-02T05:07:50.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62292025-03-02T05:07:50.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
62302025-03-02T05:07:50.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62312025-03-02T05:07:50.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62322025-03-02T05:07:50.446ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62332025-03-02T05:07:50.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62342025-03-02T05:07:50.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62352025-03-02T05:07:50.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62362025-03-02T05:07:50.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62372025-03-02T05:07:50.451ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
62382025-03-02T05:07:50.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62392025-03-02T05:07:50.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62402025-03-02T05:07:50.463ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
62412025-03-02T05:07:50.463ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.210629ms iteration = 72 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
62422025-03-02T05:07:50.476ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.906826ms iteration = 72 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
62432025-03-02T05:07:52.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 260 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = cfeb817d-a749-43ba-87f1-44103181b114 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
62442025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 25 reason = Timeout
62452025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 13 reason = Timeout
62462025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 13 reason = Timeout
62472025-03-02T05:08:00.438ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-02 04:58:00.438512739 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
62482025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
62492025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 13 reason = Timeout
62502025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
62512025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
62522025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 15 reason = Timeout
62532025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
62542025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 13 reason = Timeout
62552025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 25 reason = Timeout
62562025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 13 reason = Timeout
62572025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
62582025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
62592025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 25 reason = Timeout
62602025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
62612025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
62622025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 15 reason = Timeout
62632025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 13 reason = Timeout
62642025-03-02T05:08:00.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
62652025-03-02T05:08:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
62662025-03-02T05:08:00.439ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3bf56dce-8353-46bb-a2f0-982fbd876e0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("742da26a-88eb-42ac-9329-e125fe787858.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 3809, tv_nsec: 667064209 } })
62672025-03-02T05:08:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
62682025-03-02T05:08:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
62692025-03-02T05:08:00.439ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4a50e69a-1735-4503-96cb-55c6cc854815.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 3809, tv_nsec: 667157511 } })
62702025-03-02T05:08:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
62712025-03-02T05:08:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
62722025-03-02T05:08:00.439ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
62732025-03-02T05:08:00.439ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
62742025-03-02T05:08:00.439ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 997.603\u{b5}s iteration = 13 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
62752025-03-02T05:08:00.439ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
62762025-03-02T05:08:00.439ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 949.302\u{b5}s iteration = 13 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
62772025-03-02T05:08:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
62782025-03-02T05:08:00.441ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62792025-03-02T05:08:00.441ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 27 reason = Timeout
62802025-03-02T05:08:00.441ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
62812025-03-02T05:08:00.441ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 25 reason = Timeout
62822025-03-02T05:08:00.441ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 6 reason = Timeout
62832025-03-02T05:08:00.441ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
62842025-03-02T05:08:00.441ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 14 reason = Timeout
62852025-03-02T05:08:00.441ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca
62862025-03-02T05:08:00.441ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.92\u{b5}s iteration = 14 status = {"enabled":false,"target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca"}
62872025-03-02T05:08:00.441ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
62882025-03-02T05:08:00.441ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 13 reason = Timeout
62892025-03-02T05:08:00.441ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
62902025-03-02T05:08:00.441ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 73 reason = Timeout
62912025-03-02T05:08:00.441ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
62922025-03-02T05:08:00.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62932025-03-02T05:08:00.441ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 25 reason = Timeout
62942025-03-02T05:08:00.441ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
62952025-03-02T05:08:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
62962025-03-02T05:08:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
62972025-03-02T05:08:00.442ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
62982025-03-02T05:08:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
62992025-03-02T05:08:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
63002025-03-02T05:08:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
63012025-03-02T05:08:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63022025-03-02T05:08:00.442ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-02 04:58:00.438512739 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
63032025-03-02T05:08:00.442ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.739099ms iteration = 13 status = {"expiration":"2025-03-02T04:58:00.438512739Z","pruned":[]}
63042025-03-02T05:08:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
63052025-03-02T05:08:00.442ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
63062025-03-02T05:08:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
63072025-03-02T05:08:00.442ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
63082025-03-02T05:08:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
63092025-03-02T05:08:00.442ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
63102025-03-02T05:08:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
63112025-03-02T05:08:00.443ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
63122025-03-02T05:08:00.443ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
63132025-03-02T05:08:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
63142025-03-02T05:08:00.443ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 25 reason = Timeout
63152025-03-02T05:08:00.443ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63162025-03-02T05:08:00.443ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 25 reason = Timeout
63172025-03-02T05:08:00.443ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
63182025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
63192025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 13 reason = Timeout
63202025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
63212025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63222025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 25 reason = Timeout
63232025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
63242025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 25 reason = Timeout
63252025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
63262025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
63272025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 25 reason = Timeout
63282025-03-02T05:08:00.444ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
63292025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63302025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
63312025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 25 reason = Timeout
63322025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
63332025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
63342025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63352025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 6.084129ms iteration = 25 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
63362025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 13 reason = Timeout
63372025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
63382025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 13 reason = Timeout
63392025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
63402025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
63412025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 25 reason = Timeout
63422025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
63432025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
63442025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 73 reason = Timeout
63452025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
63462025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
63472025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 25 reason = Timeout
63482025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
63492025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
63502025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 25 reason = Timeout
63512025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
63522025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 25 reason = Timeout
63532025-03-02T05:08:00.444ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
63542025-03-02T05:08:00.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
63552025-03-02T05:08:00.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63562025-03-02T05:08:00.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
63572025-03-02T05:08:00.445ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
63582025-03-02T05:08:00.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
63592025-03-02T05:08:00.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
63602025-03-02T05:08:00.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
63612025-03-02T05:08:00.445ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 25 reason = Timeout
63622025-03-02T05:08:00.445ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
63632025-03-02T05:08:00.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
63642025-03-02T05:08:00.447ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3809, tv_nsec: 675296347 } })
63652025-03-02T05:08:00.447ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
63662025-03-02T05:08:00.447ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63672025-03-02T05:08:00.447ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3809, tv_nsec: 675760863 } })
63682025-03-02T05:08:00.447ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
63692025-03-02T05:08:00.448ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63702025-03-02T05:08:00.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
63712025-03-02T05:08:00.450ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
63722025-03-02T05:08:00.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
63732025-03-02T05:08:00.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
63742025-03-02T05:08:00.450ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
63752025-03-02T05:08:00.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
63762025-03-02T05:08:00.452ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63772025-03-02T05:08:00.453ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63782025-03-02T05:08:00.456ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
63792025-03-02T05:08:00.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63802025-03-02T05:08:00.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63812025-03-02T05:08:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
63822025-03-02T05:08:00.457ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
63832025-03-02T05:08:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
63842025-03-02T05:08:00.457ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63852025-03-02T05:08:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
63862025-03-02T05:08:00.459ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
63872025-03-02T05:08:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
63882025-03-02T05:08:00.459ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63892025-03-02T05:08:00.460ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
63902025-03-02T05:08:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
63912025-03-02T05:08:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
63922025-03-02T05:08:00.461ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
63932025-03-02T05:08:00.461ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 17.814372ms iteration = 13 status = {"errors":[],"found_port_ok":[]}
63942025-03-02T05:08:00.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
63952025-03-02T05:08:00.464ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63962025-03-02T05:08:00.466ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
63972025-03-02T05:08:00.467ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63982025-03-02T05:08:00.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63992025-03-02T05:08:00.473ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 28.575013ms iteration = 25 status = {"errors":[],"garbage_collect_requested":[]}
64002025-03-02T05:08:00.476ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64012025-03-02T05:08:00.479ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
64022025-03-02T05:08:00.479ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
64032025-03-02T05:08:00.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
64042025-03-02T05:08:00.487ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64052025-03-02T05:08:00.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64062025-03-02T05:08:00.511ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64072025-03-02T05:08:00.514ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
64082025-03-02T05:08:00.514ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
64092025-03-02T05:08:00.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
64102025-03-02T05:08:00.523ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64112025-03-02T05:08:00.525ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
64122025-03-02T05:08:00.525ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
64132025-03-02T05:08:00.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
64142025-03-02T05:08:00.527ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64152025-03-02T05:08:00.528ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64162025-03-02T05:08:00.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64172025-03-02T05:08:00.529ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
64182025-03-02T05:08:00.529ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
64192025-03-02T05:08:00.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
64202025-03-02T05:08:00.532ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64212025-03-02T05:08:00.532ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
64222025-03-02T05:08:00.532ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
64232025-03-02T05:08:00.532ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
64242025-03-02T05:08:00.532ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 88.152678ms iteration = 13 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
64252025-03-02T05:08:00.532ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
64262025-03-02T05:08:00.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
64272025-03-02T05:08:00.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
64282025-03-02T05:08:00.533ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64292025-03-02T05:08:00.534ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 90.471228ms iteration = 25 status = {"errors":[],"finish_invoked_ok":[]}
64302025-03-02T05:08:00.536ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
64312025-03-02T05:08:00.536ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
64322025-03-02T05:08:00.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
64332025-03-02T05:08:00.537ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
64342025-03-02T05:08:00.537ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64352025-03-02T05:08:00.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
64362025-03-02T05:08:00.538ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
64372025-03-02T05:08:00.538ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64382025-03-02T05:08:00.538ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
64392025-03-02T05:08:00.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
64402025-03-02T05:08:00.540ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64412025-03-02T05:08:00.540ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64422025-03-02T05:08:00.540ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
64432025-03-02T05:08:00.540ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
64442025-03-02T05:08:00.540ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
64452025-03-02T05:08:00.540ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
64462025-03-02T05:08:00.540ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64472025-03-02T05:08:00.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64482025-03-02T05:08:00.541ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 96.158241ms iteration = 25 status = {"errors":[],"requests_created_ok":[]}
64492025-03-02T05:08:00.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
64502025-03-02T05:08:00.542ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
64512025-03-02T05:08:00.542ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
64522025-03-02T05:08:00.542ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64532025-03-02T05:08:00.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
64542025-03-02T05:08:00.544ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 102.332152ms iteration = 6 status = {"nsuccess":5}
64552025-03-02T05:08:00.544ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
64562025-03-02T05:08:00.544ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64572025-03-02T05:08:00.544ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64582025-03-02T05:08:00.545ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
64592025-03-02T05:08:00.546ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
64602025-03-02T05:08:00.546ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64612025-03-02T05:08:00.546ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64622025-03-02T05:08:00.549ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 105.071888ms iteration = 73 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
64632025-03-02T05:08:00.551ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64642025-03-02T05:08:00.551ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64652025-03-02T05:08:00.552ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
64662025-03-02T05:08:00.552ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
64672025-03-02T05:08:00.552ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
64682025-03-02T05:08:00.552ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
64692025-03-02T05:08:00.553ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3809, tv_nsec: 781169285 } })
64702025-03-02T05:08:00.553ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
64712025-03-02T05:08:00.553ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
64722025-03-02T05:08:00.553ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 111.498372ms iteration = 13 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
64732025-03-02T05:08:00.553ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
64742025-03-02T05:08:00.567ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
64752025-03-02T05:08:00.567ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 125.788478ms iteration = 73 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
64762025-03-02T05:08:00.600ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
64772025-03-02T05:08:00.600ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
64782025-03-02T05:08:00.601ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "95bd996d-0a50-41ef-8096-0ffb46e56d9a", "content-length": "26", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} })
64792025-03-02T05:08:00.601ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
64802025-03-02T05:08:00.603ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64812025-03-02T05:08:00.604ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
64822025-03-02T05:08:00.604ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
64832025-03-02T05:08:00.604ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
64842025-03-02T05:08:00.604ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
64852025-03-02T05:08:00.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
64862025-03-02T05:08:00.606ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
64872025-03-02T05:08:00.606ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
64882025-03-02T05:08:00.606ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
64892025-03-02T05:08:00.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
64902025-03-02T05:08:00.606ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
64912025-03-02T05:08:00.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
64922025-03-02T05:08:00.607ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
64932025-03-02T05:08:00.607ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
64942025-03-02T05:08:00.607ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
64952025-03-02T05:08:00.607ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
64962025-03-02T05:08:00.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
64972025-03-02T05:08:00.608ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64982025-03-02T05:08:00.610ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
64992025-03-02T05:08:00.610ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
65002025-03-02T05:08:00.610ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 165.948601ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
65012025-03-02T05:08:00.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
65022025-03-02T05:08:00.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
65032025-03-02T05:08:00.612ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
65042025-03-02T05:08:00.612ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
65052025-03-02T05:08:00.612ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65062025-03-02T05:08:00.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
65072025-03-02T05:08:00.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65082025-03-02T05:08:00.612ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
65092025-03-02T05:08:00.612ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 167.190806ms iteration = 25 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
65102025-03-02T05:08:00.614ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65112025-03-02T05:08:00.616ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
65122025-03-02T05:08:00.616ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
65132025-03-02T05:08:00.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
65142025-03-02T05:08:00.618ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 173.854214ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
65152025-03-02T05:08:00.620ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
65162025-03-02T05:08:00.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
65172025-03-02T05:08:00.621ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
65182025-03-02T05:08:00.621ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
65192025-03-02T05:08:00.621ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 176.869194ms iteration = 25 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
65202025-03-02T05:08:00.644ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 199.557768ms iteration = 25 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
65212025-03-02T05:08:00.644ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65222025-03-02T05:08:00.647ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
65232025-03-02T05:08:00.647ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
65242025-03-02T05:08:00.647ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
65252025-03-02T05:08:00.668ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
65262025-03-02T05:08:00.668ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
65272025-03-02T05:08:00.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
65282025-03-02T05:08:00.701ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
65292025-03-02T05:08:00.701ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
65302025-03-02T05:08:00.701ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff78948b-f71f-4843-9c4f-b81c3bb5c08b", "content-length": "26", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} })
65312025-03-02T05:08:00.701ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
65322025-03-02T05:08:00.702ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
65332025-03-02T05:08:00.702ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 257.908778ms iteration = 13 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
65342025-03-02T05:08:00.702ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
65352025-03-02T05:08:00.705ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 264.084739ms iteration = 27 status = {"physical_disks_added":0}
65362025-03-02T05:08:00.710ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
65372025-03-02T05:08:00.710ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
65382025-03-02T05:08:00.710ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bd6870c-c3db-40c1-b889-f49b9d4a9c5a", "content-length": "26", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} })
65392025-03-02T05:08:00.710ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
65402025-03-02T05:08:00.712ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
65412025-03-02T05:08:00.740ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
65422025-03-02T05:08:00.766ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 328.301805ms iteration = 25 status = {}
65432025-03-02T05:08:00.832ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
65442025-03-02T05:08:00.842ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
65452025-03-02T05:08:00.842ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
65462025-03-02T05:08:00.843ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a5fe256-3ac1-4005-a108-a5166c047bbb", "content-length": "372", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} })
65472025-03-02T05:08:00.843ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65482025-03-02T05:08:00.843ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65492025-03-02T05:08:00.846ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65502025-03-02T05:08:00.846ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65512025-03-02T05:08:00.846ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65522025-03-02T05:08:00.848ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
65532025-03-02T05:08:00.848ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
65542025-03-02T05:08:00.848ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
65552025-03-02T05:08:00.853ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
65562025-03-02T05:08:00.854ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca9ededc-0ba2-4f2e-8b25-210c0e3ae57c", "content-length": "2", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} })
65572025-03-02T05:08:00.859ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
65582025-03-02T05:08:00.859ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
65592025-03-02T05:08:00.865ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 427.107673ms iteration = 25 status = null
65602025-03-02T05:08:00.869ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
65612025-03-02T05:08:00.869ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "36a1382d-0bc8-4711-8dd6-351bb5252636", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
65622025-03-02T05:08:00.869ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
65632025-03-02T05:08:00.872ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "e6a97101-79b8-4bb8-913f-682557c83b19", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
65642025-03-02T05:08:00.872ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
65652025-03-02T05:08:00.873ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
65662025-03-02T05:08:00.873ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 434.869044ms iteration = 15 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
65672025-03-02T05:08:00.879ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
65682025-03-02T05:08:00.879ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 0 version = 2
65692025-03-02T05:08:00.879ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
65702025-03-02T05:08:00.879ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
65712025-03-02T05:08:00.881ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65722025-03-02T05:08:00.883ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
65732025-03-02T05:08:00.883ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
65742025-03-02T05:08:00.883ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
65752025-03-02T05:08:00.884ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
65762025-03-02T05:08:00.884ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 442.436003ms iteration = 25 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
65772025-03-02T05:08:00.885ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 1 version = 2
65782025-03-02T05:08:00.885ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
65792025-03-02T05:08:00.885ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-02 04:56:26.100923 UTC dns_group = external generation = 2
65802025-03-02T05:08:00.885ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 446.951862ms iteration = 13 status = {"generation":2}
65812025-03-02T05:08:00.886ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
65822025-03-02T05:08:00.886ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
65832025-03-02T05:08:00.886ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
65842025-03-02T05:08:00.886ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
65852025-03-02T05:08:00.886ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65862025-03-02T05:08:00.887ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65872025-03-02T05:08:00.889ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
65882025-03-02T05:08:00.889ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65892025-03-02T05:08:00.889ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65902025-03-02T05:08:00.889ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-02T04:55:57.280988Z, time_modified: 2025-03-02T04:55:57.280988Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }) }
65912025-03-02T05:08:00.889ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 451.214426ms iteration = 13 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]}},"default_endpoint":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
65922025-03-02T05:08:00.890ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
65932025-03-02T05:08:00.890ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
65942025-03-02T05:08:00.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
65952025-03-02T05:08:00.890ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 449.310711ms iteration = 25 status = 0
65962025-03-02T05:08:00.892ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
65972025-03-02T05:08:00.892ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
65982025-03-02T05:08:00.892ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-02 04:56:25.487890 UTC dns_group = internal generation = 1
65992025-03-02T05:08:00.892ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 454.396589ms iteration = 13 status = {"generation":1}
66002025-03-02T05:08:00.895ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
66012025-03-02T05:08:00.895ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "1324c798-68f5-4ccf-ac1a-46a4c3657cdf", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
66022025-03-02T05:08:00.895ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
66032025-03-02T05:08:00.898ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "a65630d7-557e-4a66-9934-a8f4c0de415c", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
66042025-03-02T05:08:00.898ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
66052025-03-02T05:08:00.898ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
66062025-03-02T05:08:00.898ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
66072025-03-02T05:08:00.898ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "7e6c7dd3-df55-432c-b07f-8cfb321f20de", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} })
66082025-03-02T05:08:00.898ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
66092025-03-02T05:08:00.898ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
66102025-03-02T05:08:00.900ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
66112025-03-02T05:08:00.900ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
66122025-03-02T05:08:00.900ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "ec2ccbee-bdf1-442c-95df-837b470251d0", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
66132025-03-02T05:08:00.900ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
66142025-03-02T05:08:00.902ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
66152025-03-02T05:08:00.902ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 464.009563ms iteration = 15 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
66162025-03-02T05:08:00.903ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
66172025-03-02T05:08:00.903ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
66182025-03-02T05:08:00.906ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
66192025-03-02T05:08:00.906ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
66202025-03-02T05:08:00.906ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "41be0f50-2059-43de-ba6f-16b957a0f12d", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} })
66212025-03-02T05:08:00.908ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 463.823821ms iteration = 25 status = {}
66222025-03-02T05:08:00.947ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66232025-03-02T05:08:00.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66242025-03-02T05:08:00.949ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
66252025-03-02T05:08:00.949ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66262025-03-02T05:08:00.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66272025-03-02T05:08:00.950ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
66282025-03-02T05:08:00.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
66292025-03-02T05:08:00.951ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66302025-03-02T05:08:00.952ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
66312025-03-02T05:08:00.952ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
66322025-03-02T05:08:00.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
66332025-03-02T05:08:00.954ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66342025-03-02T05:08:00.956ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
66352025-03-02T05:08:00.956ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
66362025-03-02T05:08:00.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
66372025-03-02T05:08:00.960ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
66382025-03-02T05:08:00.960ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "2849e1d5-fd08-4a93-92cc-5c5f73f83bf2", "content-length": "134", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} })
66392025-03-02T05:08:00.960ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
66402025-03-02T05:08:00.960ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
66412025-03-02T05:08:00.960ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
66422025-03-02T05:08:00.960ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
66432025-03-02T05:08:00.960ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66442025-03-02T05:08:00.960ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66452025-03-02T05:08:00.960ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66462025-03-02T05:08:00.960ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
66472025-03-02T05:08:00.961ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66482025-03-02T05:08:00.961ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
66492025-03-02T05:08:00.961ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
66502025-03-02T05:08:00.961ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66512025-03-02T05:08:00.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66522025-03-02T05:08:00.963ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
66532025-03-02T05:08:00.963ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "81b8f10b-e123-4e77-b5ca-0488bb53bb47", "content-length": "2", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} })
66542025-03-02T05:08:00.963ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
66552025-03-02T05:08:00.963ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ec7f8ec-4358-4405-99f7-105b3b616d7a", "content-length": "52", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} })
66562025-03-02T05:08:00.963ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
66572025-03-02T05:08:00.963ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
66582025-03-02T05:08:00.989ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
66592025-03-02T05:08:01.058ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "fcad8e86-4939-408a-bba7-06badc9d431c", "date": "Sun, 02 Mar 2025 05:08:00 GMT"} })
66602025-03-02T05:08:01.058ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
66612025-03-02T05:08:01.058ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
66622025-03-02T05:08:01.059ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "6984f0f5-dfcb-48a7-8099-921bbb3d9df4", "content-length": "682", "date": "Sun, 02 Mar 2025 05:08:01 GMT"} })
66632025-03-02T05:08:01.059ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66642025-03-02T05:08:01.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66652025-03-02T05:08:01.059ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "06563fea-51ca-4c33-8984-b15e0b4e3d29", "content-length": "691", "date": "Sun, 02 Mar 2025 05:08:01 GMT"} })
66662025-03-02T05:08:01.059ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
66672025-03-02T05:08:01.059ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
66682025-03-02T05:08:01.059ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
66692025-03-02T05:08:01.061ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 617.27707ms iteration = 25 status = {}
66702025-03-02T05:08:01.063ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66712025-03-02T05:08:01.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66722025-03-02T05:08:01.067ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66732025-03-02T05:08:01.067ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66742025-03-02T05:08:01.069ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66752025-03-02T05:08:01.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66762025-03-02T05:08:01.077ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 633.293569ms iteration = 25 status = {}
66772025-03-02T05:08:02.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 292 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 0baee4a9-198d-4d27-99c4-1ae27fd42f45 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
66782025-03-02T05:08:10.441ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 74 reason = Timeout
66792025-03-02T05:08:10.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66802025-03-02T05:08:10.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66812025-03-02T05:08:10.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 74 reason = Timeout
66822025-03-02T05:08:10.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66832025-03-02T05:08:10.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66842025-03-02T05:08:10.446ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66852025-03-02T05:08:10.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66862025-03-02T05:08:10.471ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66872025-03-02T05:08:10.473ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
66882025-03-02T05:08:10.474ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66892025-03-02T05:08:10.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66902025-03-02T05:08:10.476ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66912025-03-02T05:08:10.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66922025-03-02T05:08:10.477ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.899168ms iteration = 74 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
66932025-03-02T05:08:10.477ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66942025-03-02T05:08:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
66952025-03-02T05:08:10.480ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66962025-03-02T05:08:10.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66972025-03-02T05:08:10.491ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
66982025-03-02T05:08:10.491ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 49.466775ms iteration = 74 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
66992025-03-02T05:08:12.896ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:44175
67002025-03-02T05:08:12.896ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44175 req_id = 71e9e39d-30eb-4580-9f26-645d05d4dd6d roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
67012025-03-02T05:08:12.897ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44175 req_id = 71e9e39d-30eb-4580-9f26-645d05d4dd6d resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
67022025-03-02T05:08:12.900ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3398 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44175 req_id = 71e9e39d-30eb-4580-9f26-645d05d4dd6d response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
67032025-03-02T05:08:12.900ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44175 req_id = e43368bd-24d9-4cce-a7b8-900406cf7085 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
67042025-03-02T05:08:12.900ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44175 req_id = e43368bd-24d9-4cce-a7b8-900406cf7085 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
67052025-03-02T05:08:12.902ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2599 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44175 req_id = e43368bd-24d9-4cce-a7b8-900406cf7085 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
67062025-03-02T05:08:12.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 291 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 7f8af4cb-295b-40ff-b911-96cc7a8c01d2 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
67072025-03-02T05:08:20.441ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 75 reason = Timeout
67082025-03-02T05:08:20.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67092025-03-02T05:08:20.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67102025-03-02T05:08:20.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 75 reason = Timeout
67112025-03-02T05:08:20.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67122025-03-02T05:08:20.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67132025-03-02T05:08:20.446ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67142025-03-02T05:08:20.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67152025-03-02T05:08:20.473ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67162025-03-02T05:08:20.475ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
67172025-03-02T05:08:20.475ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67182025-03-02T05:08:20.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67192025-03-02T05:08:20.477ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67202025-03-02T05:08:20.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67212025-03-02T05:08:20.479ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67222025-03-02T05:08:20.481ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
67232025-03-02T05:08:20.481ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67242025-03-02T05:08:20.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67252025-03-02T05:08:20.482ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 38.37357ms iteration = 75 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
67262025-03-02T05:08:20.494ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
67272025-03-02T05:08:20.494ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 52.735107ms iteration = 75 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
67282025-03-02T05:08:22.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 291 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 2fcab743-ab70-4be8-bde7-557aa48b5bde response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
67292025-03-02T05:08:30.438ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 26 reason = Timeout
67302025-03-02T05:08:30.438ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 26 reason = Timeout
67312025-03-02T05:08:30.438ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 26 reason = Timeout
67322025-03-02T05:08:30.438ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
67332025-03-02T05:08:30.438ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
67342025-03-02T05:08:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
67352025-03-02T05:08:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
67362025-03-02T05:08:30.441ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 26 reason = Timeout
67372025-03-02T05:08:30.441ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 76 reason = Timeout
67382025-03-02T05:08:30.441ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
67392025-03-02T05:08:30.441ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 26 reason = Timeout
67402025-03-02T05:08:30.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67412025-03-02T05:08:30.441ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 28 reason = Timeout
67422025-03-02T05:08:30.441ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
67432025-03-02T05:08:30.441ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
67442025-03-02T05:08:30.441ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3839, tv_nsec: 669482013 } })
67452025-03-02T05:08:30.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
67462025-03-02T05:08:30.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
67472025-03-02T05:08:30.441ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3839, tv_nsec: 669575314 } })
67482025-03-02T05:08:30.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
67492025-03-02T05:08:30.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67502025-03-02T05:08:30.441ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
67512025-03-02T05:08:30.442ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.88788ms iteration = 26 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
67522025-03-02T05:08:30.442ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
67532025-03-02T05:08:30.443ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67542025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67552025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 26 reason = Timeout
67562025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
67572025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 26 reason = Timeout
67582025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67592025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 26 reason = Timeout
67602025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
67612025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
67622025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 26 reason = Timeout
67632025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
67642025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 26 reason = Timeout
67652025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
67662025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 26 reason = Timeout
67672025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
67682025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 26 reason = Timeout
67692025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
67702025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 26 reason = Timeout
67712025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
67722025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
67732025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 76 reason = Timeout
67742025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67752025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 26 reason = Timeout
67762025-03-02T05:08:30.444ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
67772025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
67782025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 26 reason = Timeout
67792025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
67802025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67812025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
67822025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
67832025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
67842025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
67852025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
67862025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
67872025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
67882025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67892025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
67902025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
67912025-03-02T05:08:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
67922025-03-02T05:08:30.445ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 26 reason = Timeout
67932025-03-02T05:08:30.445ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
67942025-03-02T05:08:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
67952025-03-02T05:08:30.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
67962025-03-02T05:08:30.447ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
67972025-03-02T05:08:30.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
67982025-03-02T05:08:30.447ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
67992025-03-02T05:08:30.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68002025-03-02T05:08:30.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68012025-03-02T05:08:30.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
68022025-03-02T05:08:30.447ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
68032025-03-02T05:08:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68042025-03-02T05:08:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
68052025-03-02T05:08:30.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
68062025-03-02T05:08:30.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
68072025-03-02T05:08:30.457ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68082025-03-02T05:08:30.459ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
68092025-03-02T05:08:30.460ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
68102025-03-02T05:08:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
68112025-03-02T05:08:30.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
68122025-03-02T05:08:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
68132025-03-02T05:08:30.462ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68142025-03-02T05:08:30.465ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
68152025-03-02T05:08:30.465ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68162025-03-02T05:08:30.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68172025-03-02T05:08:30.468ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68182025-03-02T05:08:30.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68192025-03-02T05:08:30.469ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
68202025-03-02T05:08:30.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68212025-03-02T05:08:30.470ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
68222025-03-02T05:08:30.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
68232025-03-02T05:08:30.470ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68242025-03-02T05:08:30.471ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3839, tv_nsec: 699735217 } })
68252025-03-02T05:08:30.472ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
68262025-03-02T05:08:30.473ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
68272025-03-02T05:08:30.473ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68282025-03-02T05:08:30.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68292025-03-02T05:08:30.512ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 68.655275ms iteration = 26 status = {"errors":[],"finish_invoked_ok":[]}
68302025-03-02T05:08:30.513ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68312025-03-02T05:08:30.516ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
68322025-03-02T05:08:30.516ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
68332025-03-02T05:08:30.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
68342025-03-02T05:08:30.518ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
68352025-03-02T05:08:30.518ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
68362025-03-02T05:08:30.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
68372025-03-02T05:08:30.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
68382025-03-02T05:08:30.519ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
68392025-03-02T05:08:30.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68402025-03-02T05:08:30.520ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68412025-03-02T05:08:30.522ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 78.204328ms iteration = 76 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
68422025-03-02T05:08:30.522ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68432025-03-02T05:08:30.523ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
68442025-03-02T05:08:30.523ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
68452025-03-02T05:08:30.523ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
68462025-03-02T05:08:30.523ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
68472025-03-02T05:08:30.523ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
68482025-03-02T05:08:30.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
68492025-03-02T05:08:30.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68502025-03-02T05:08:30.524ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 79.894861ms iteration = 26 status = {"errors":[],"garbage_collect_requested":[]}
68512025-03-02T05:08:30.525ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
68522025-03-02T05:08:30.525ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
68532025-03-02T05:08:30.525ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f3f40fe-72b3-4be7-a366-d7e13f834a50", "content-length": "26", "date": "Sun, 02 Mar 2025 05:08:30 GMT"} })
68542025-03-02T05:08:30.525ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
68552025-03-02T05:08:30.526ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 81.801915ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
68562025-03-02T05:08:30.526ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
68572025-03-02T05:08:30.526ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
68582025-03-02T05:08:30.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68592025-03-02T05:08:30.530ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
68602025-03-02T05:08:30.530ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
68612025-03-02T05:08:30.530ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
68622025-03-02T05:08:30.530ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
68632025-03-02T05:08:30.530ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
68642025-03-02T05:08:30.530ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
68652025-03-02T05:08:30.530ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 89.629167ms iteration = 26 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
68662025-03-02T05:08:30.531ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68672025-03-02T05:08:30.533ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
68682025-03-02T05:08:30.533ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 89.545486ms iteration = 26 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
68692025-03-02T05:08:30.534ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
68702025-03-02T05:08:30.534ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
68712025-03-02T05:08:30.534ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
68722025-03-02T05:08:30.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
68732025-03-02T05:08:30.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
68742025-03-02T05:08:30.534ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 89.242052ms iteration = 26 status = {"errors":[],"requests_created_ok":[]}
68752025-03-02T05:08:30.534ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
68762025-03-02T05:08:30.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
68772025-03-02T05:08:30.544ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68782025-03-02T05:08:30.544ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68792025-03-02T05:08:30.546ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68802025-03-02T05:08:30.546ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68812025-03-02T05:08:30.547ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
68822025-03-02T05:08:30.547ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 106.70686ms iteration = 76 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
68832025-03-02T05:08:30.548ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68842025-03-02T05:08:30.550ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
68852025-03-02T05:08:30.550ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68862025-03-02T05:08:30.551ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68872025-03-02T05:08:30.552ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 111.716005ms iteration = 26 status = 0
68882025-03-02T05:08:30.554ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
68892025-03-02T05:08:30.554ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
68902025-03-02T05:08:30.554ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1436b8c1-b6c8-45e7-b39f-123faebfa009", "content-length": "26", "date": "Sun, 02 Mar 2025 05:08:30 GMT"} })
68912025-03-02T05:08:30.554ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
68922025-03-02T05:08:30.556ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
68932025-03-02T05:08:30.566ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 122.716758ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
68942025-03-02T05:08:30.571ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
68952025-03-02T05:08:30.571ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
68962025-03-02T05:08:30.572ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
68972025-03-02T05:08:30.572ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
68982025-03-02T05:08:30.573ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3420c33f-1198-45fa-9189-5c6896769e09", "content-length": "26", "date": "Sun, 02 Mar 2025 05:08:30 GMT"} })
68992025-03-02T05:08:30.573ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
69002025-03-02T05:08:30.573ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
69012025-03-02T05:08:30.573ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
69022025-03-02T05:08:30.573ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 129.416985ms iteration = 26 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
69032025-03-02T05:08:30.575ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
69042025-03-02T05:08:30.604ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 159.991924ms iteration = 26 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
69052025-03-02T05:08:30.636ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 195.316144ms iteration = 28 status = {"physical_disks_added":0}
69062025-03-02T05:08:30.650ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
69072025-03-02T05:08:30.651ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
69082025-03-02T05:08:30.653ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69092025-03-02T05:08:30.654ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
69102025-03-02T05:08:30.654ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
69112025-03-02T05:08:30.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
69122025-03-02T05:08:30.656ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69132025-03-02T05:08:30.657ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
69142025-03-02T05:08:30.657ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
69152025-03-02T05:08:30.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
69162025-03-02T05:08:30.707ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
69172025-03-02T05:08:30.707ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
69182025-03-02T05:08:30.707ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfc79ba5-4924-48c4-87db-8cef21a326af", "content-length": "372", "date": "Sun, 02 Mar 2025 05:08:30 GMT"} })
69192025-03-02T05:08:30.707ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69202025-03-02T05:08:30.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69212025-03-02T05:08:30.749ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
69222025-03-02T05:08:30.749ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
69232025-03-02T05:08:30.749ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "21510ccc-20e4-4006-8714-3f72f8479a6f", "date": "Sun, 02 Mar 2025 05:08:30 GMT"} })
69242025-03-02T05:08:30.749ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69252025-03-02T05:08:30.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69262025-03-02T05:08:30.752ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69272025-03-02T05:08:30.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69282025-03-02T05:08:30.755ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69292025-03-02T05:08:30.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69302025-03-02T05:08:30.758ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
69312025-03-02T05:08:30.759ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
69322025-03-02T05:08:30.759ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "19c8e6b5-160a-42ee-8b1b-86c72fc10107", "date": "Sun, 02 Mar 2025 05:08:30 GMT"} })
69332025-03-02T05:08:30.760ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 316.479452ms iteration = 26 status = {}
69342025-03-02T05:08:30.815ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 377.601118ms iteration = 26 status = {}
69352025-03-02T05:08:30.863ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
69362025-03-02T05:08:30.864ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9e4c3dc-0859-4bd9-b9d2-0209ec586667", "content-length": "2", "date": "Sun, 02 Mar 2025 05:08:30 GMT"} })
69372025-03-02T05:08:30.865ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
69382025-03-02T05:08:30.866ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
69392025-03-02T05:08:30.874ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
69402025-03-02T05:08:30.876ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 437.816283ms iteration = 26 status = null
69412025-03-02T05:08:30.882ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "8eb9adb2-e13f-491e-9941-f88d29cafe7d", "content-length": "691", "date": "Sun, 02 Mar 2025 05:08:30 GMT"} })
69422025-03-02T05:08:30.882ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
69432025-03-02T05:08:30.882ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
69442025-03-02T05:08:30.882ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
69452025-03-02T05:08:30.883ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69462025-03-02T05:08:30.883ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69472025-03-02T05:08:30.885ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 440.855553ms iteration = 26 status = {}
69482025-03-02T05:08:30.885ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
69492025-03-02T05:08:30.885ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69502025-03-02T05:08:30.885ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69512025-03-02T05:08:30.895ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
69522025-03-02T05:08:30.895ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa9e7681-3483-4511-b7be-0ae2a8889807", "content-length": "134", "date": "Sun, 02 Mar 2025 05:08:30 GMT"} })
69532025-03-02T05:08:30.895ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
69542025-03-02T05:08:30.895ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
69552025-03-02T05:08:30.896ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
69562025-03-02T05:08:30.896ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
69572025-03-02T05:08:30.896ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
69582025-03-02T05:08:30.896ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
69592025-03-02T05:08:30.896ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
69602025-03-02T05:08:30.896ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
69612025-03-02T05:08:30.896ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
69622025-03-02T05:08:30.896ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
69632025-03-02T05:08:30.896ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
69642025-03-02T05:08:30.896ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69652025-03-02T05:08:30.896ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69662025-03-02T05:08:30.898ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
69672025-03-02T05:08:30.898ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "17e4d69c-35cb-4b4d-9a14-576f3fb97064", "content-length": "2", "date": "Sun, 02 Mar 2025 05:08:30 GMT"} })
69682025-03-02T05:08:30.898ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
69692025-03-02T05:08:30.898ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae236dd9-41d1-4dce-850a-767b3cd6f55e", "content-length": "52", "date": "Sun, 02 Mar 2025 05:08:30 GMT"} })
69702025-03-02T05:08:30.898ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
69712025-03-02T05:08:30.898ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
69722025-03-02T05:08:30.988ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "538376fa-3bec-4903-8857-daf20b1380f7", "date": "Sun, 02 Mar 2025 05:08:30 GMT"} })
69732025-03-02T05:08:30.988ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
69742025-03-02T05:08:30.988ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
69752025-03-02T05:08:30.989ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "add67443-1cf8-4035-91e5-d319890bc0d4", "content-length": "682", "date": "Sun, 02 Mar 2025 05:08:30 GMT"} })
69762025-03-02T05:08:30.989ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69772025-03-02T05:08:30.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69782025-03-02T05:08:30.993ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69792025-03-02T05:08:30.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69802025-03-02T05:08:30.996ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69812025-03-02T05:08:30.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69822025-03-02T05:08:30.998ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
69832025-03-02T05:08:30.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
69842025-03-02T05:08:31.006ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 562.01237ms iteration = 26 status = {}
69852025-03-02T05:08:32.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 247 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 6ca45875-942a-4146-9fe1-e13b4158e680 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
69862025-03-02T05:08:40.441ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 77 reason = Timeout
69872025-03-02T05:08:40.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69882025-03-02T05:08:40.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69892025-03-02T05:08:40.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69902025-03-02T05:08:40.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 77 reason = Timeout
69912025-03-02T05:08:40.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
69922025-03-02T05:08:40.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
69932025-03-02T05:08:40.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
69942025-03-02T05:08:40.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69952025-03-02T05:08:40.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69962025-03-02T05:08:40.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69972025-03-02T05:08:40.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69982025-03-02T05:08:40.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69992025-03-02T05:08:40.451ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
70002025-03-02T05:08:40.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70012025-03-02T05:08:40.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70022025-03-02T05:08:40.467ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
70032025-03-02T05:08:40.467ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.776746ms iteration = 77 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
70042025-03-02T05:08:40.481ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
70052025-03-02T05:08:40.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
70062025-03-02T05:08:40.483ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 38.929837ms iteration = 77 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
70072025-03-02T05:08:42.897ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44175 req_id = 7f69f2da-5d10-4aea-9e27-7d69e48f6de9 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
70082025-03-02T05:08:42.897ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44175 req_id = 7f69f2da-5d10-4aea-9e27-7d69e48f6de9 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
70092025-03-02T05:08:42.900ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3465 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44175 req_id = 7f69f2da-5d10-4aea-9e27-7d69e48f6de9 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
70102025-03-02T05:08:42.900ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44175 req_id = 507908be-b410-4c4b-8a43-3142df82412c roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
70112025-03-02T05:08:42.901ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44175 req_id = 507908be-b410-4c4b-8a43-3142df82412c resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
70122025-03-02T05:08:42.903ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2401 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44175 req_id = 507908be-b410-4c4b-8a43-3142df82412c response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
70132025-03-02T05:08:42.903ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 233 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 4656a45f-066e-4684-89cd-2fadbea15b69 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
70142025-03-02T05:08:50.441ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 78 reason = Timeout
70152025-03-02T05:08:50.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70162025-03-02T05:08:50.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70172025-03-02T05:08:50.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70182025-03-02T05:08:50.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 78 reason = Timeout
70192025-03-02T05:08:50.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
70202025-03-02T05:08:50.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
70212025-03-02T05:08:50.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
70222025-03-02T05:08:50.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70232025-03-02T05:08:50.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70242025-03-02T05:08:50.446ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
70252025-03-02T05:08:50.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
70262025-03-02T05:08:50.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70272025-03-02T05:08:50.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70282025-03-02T05:08:50.448ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.681858ms iteration = 78 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
70292025-03-02T05:08:50.450ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70302025-03-02T05:08:50.452ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
70312025-03-02T05:08:50.452ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70322025-03-02T05:08:50.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70332025-03-02T05:08:50.462ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
70342025-03-02T05:08:50.462ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.510697ms iteration = 78 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
70352025-03-02T05:08:52.904ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 306 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 836dde79-0bb2-4ab5-8613-e32666cb6a65 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
70362025-03-02T05:09:00.438ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 16 reason = Timeout
70372025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 27 reason = Timeout
70382025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 14 reason = Timeout
70392025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
70402025-03-02T05:09:00.439ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-02 04:59:00.439005254 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
70412025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
70422025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 14 reason = Timeout
70432025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 27 reason = Timeout
70442025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
70452025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 14 reason = Timeout
70462025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 14 reason = Timeout
70472025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
70482025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 16 reason = Timeout
70492025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 14 reason = Timeout
70502025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
70512025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 27 reason = Timeout
70522025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
70532025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 14 reason = Timeout
70542025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
70552025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
70562025-03-02T05:09:00.439ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4a50e69a-1735-4503-96cb-55c6cc854815.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c54e30d8-f5a2-495b-b2d8-a78319addcd7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 3869, tv_nsec: 667182045 } })
70572025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
70582025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
70592025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
70602025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
70612025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
70622025-03-02T05:09:00.439ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3bf56dce-8353-46bb-a2f0-982fbd876e0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("742da26a-88eb-42ac-9329-e125fe787858.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7b1d175f-bda6-4078-98b5-1cf73342bf0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 3869, tv_nsec: 667337767 } })
70632025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
70642025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
70652025-03-02T05:09:00.439ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 867.471\u{b5}s iteration = 14 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
70662025-03-02T05:09:00.440ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
70672025-03-02T05:09:00.440ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.080194ms iteration = 14 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
70682025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70692025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 29 reason = Timeout
70702025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 79 reason = Timeout
70712025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
70722025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 14 reason = Timeout
70732025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
70742025-03-02T05:09:00.441ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
70752025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 15 reason = Timeout
70762025-03-02T05:09:00.441ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca
70772025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
70782025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 8.82\u{b5}s iteration = 15 status = {"enabled":false,"target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca"}
70792025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 27 reason = Timeout
70802025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 27 reason = Timeout
70812025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
70822025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
70832025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
70842025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
70852025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
70862025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
70872025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
70882025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
70892025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
70902025-03-02T05:09:00.441ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3869, tv_nsec: 669513405 } })
70912025-03-02T05:09:00.441ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
70922025-03-02T05:09:00.442ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
70932025-03-02T05:09:00.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
70942025-03-02T05:09:00.442ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.575866ms iteration = 27 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
70952025-03-02T05:09:00.442ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
70962025-03-02T05:09:00.442ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3869, tv_nsec: 670589780 } })
70972025-03-02T05:09:00.442ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
70982025-03-02T05:09:00.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
70992025-03-02T05:09:00.443ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
71002025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 27 reason = Timeout
71012025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
71022025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 27 reason = Timeout
71032025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 14 reason = Timeout
71042025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
71052025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
71062025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 79 reason = Timeout
71072025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71082025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 14 reason = Timeout
71092025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
71102025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 27 reason = Timeout
71112025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71122025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 27 reason = Timeout
71132025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
71142025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 27 reason = Timeout
71152025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
71162025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 27 reason = Timeout
71172025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
71182025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 27 reason = Timeout
71192025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
71202025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 27 reason = Timeout
71212025-03-02T05:09:00.444ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
71222025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
71232025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 14 reason = Timeout
71242025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
71252025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 27 reason = Timeout
71262025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
71272025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 27 reason = Timeout
71282025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
71292025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
71302025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
71312025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
71322025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71332025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
71342025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71352025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
71362025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
71372025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
71382025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
71392025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
71402025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
71412025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
71422025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
71432025-03-02T05:09:00.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
71442025-03-02T05:09:00.445ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71452025-03-02T05:09:00.445ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-02 04:59:00.439005254 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
71462025-03-02T05:09:00.445ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 6.300353ms iteration = 14 status = {"expiration":"2025-03-02T04:59:00.439005254Z","pruned":[]}
71472025-03-02T05:09:00.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71482025-03-02T05:09:00.445ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 27 reason = Timeout
71492025-03-02T05:09:00.445ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
71502025-03-02T05:09:00.445ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71512025-03-02T05:09:00.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
71522025-03-02T05:09:00.446ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71532025-03-02T05:09:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
71542025-03-02T05:09:00.448ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
71552025-03-02T05:09:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
71562025-03-02T05:09:00.448ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
71572025-03-02T05:09:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
71582025-03-02T05:09:00.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
71592025-03-02T05:09:00.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
71602025-03-02T05:09:00.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71612025-03-02T05:09:00.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71622025-03-02T05:09:00.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
71632025-03-02T05:09:00.449ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
71642025-03-02T05:09:00.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
71652025-03-02T05:09:00.450ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71662025-03-02T05:09:00.451ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71672025-03-02T05:09:00.452ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 8.221267ms iteration = 14 status = {"errors":[],"found_port_ok":[]}
71682025-03-02T05:09:00.453ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
71692025-03-02T05:09:00.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
71702025-03-02T05:09:00.454ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
71712025-03-02T05:09:00.455ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
71722025-03-02T05:09:00.456ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71732025-03-02T05:09:00.456ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
71742025-03-02T05:09:00.456ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
71752025-03-02T05:09:00.456ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
71762025-03-02T05:09:00.456ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
71772025-03-02T05:09:00.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71782025-03-02T05:09:00.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
71792025-03-02T05:09:00.457ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71802025-03-02T05:09:00.458ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 14.197725ms iteration = 27 status = {"errors":[],"finish_invoked_ok":[]}
71812025-03-02T05:09:00.458ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
71822025-03-02T05:09:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
71832025-03-02T05:09:00.459ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71842025-03-02T05:09:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
71852025-03-02T05:09:00.460ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
71862025-03-02T05:09:00.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
71872025-03-02T05:09:00.461ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
71882025-03-02T05:09:00.461ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
71892025-03-02T05:09:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
71902025-03-02T05:09:00.467ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
71912025-03-02T05:09:00.468ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
71922025-03-02T05:09:00.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
71932025-03-02T05:09:00.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
71942025-03-02T05:09:00.470ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71952025-03-02T05:09:00.474ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
71962025-03-02T05:09:00.474ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
71972025-03-02T05:09:00.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
71982025-03-02T05:09:00.478ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71992025-03-02T05:09:00.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
72002025-03-02T05:09:00.484ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72012025-03-02T05:09:00.488ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
72022025-03-02T05:09:00.488ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72032025-03-02T05:09:00.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72042025-03-02T05:09:00.510ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72052025-03-02T05:09:00.513ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
72062025-03-02T05:09:00.513ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
72072025-03-02T05:09:00.513ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
72082025-03-02T05:09:00.518ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
72092025-03-02T05:09:00.518ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
72102025-03-02T05:09:00.518ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
72112025-03-02T05:09:00.518ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
72122025-03-02T05:09:00.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
72132025-03-02T05:09:00.521ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72142025-03-02T05:09:00.524ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
72152025-03-02T05:09:00.525ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
72162025-03-02T05:09:00.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
72172025-03-02T05:09:00.525ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72182025-03-02T05:09:00.528ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
72192025-03-02T05:09:00.528ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
72202025-03-02T05:09:00.528ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
72212025-03-02T05:09:00.528ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
72222025-03-02T05:09:00.528ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
72232025-03-02T05:09:00.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
72242025-03-02T05:09:00.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
72252025-03-02T05:09:00.530ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
72262025-03-02T05:09:00.530ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 86.090472ms iteration = 27 status = {"errors":[],"garbage_collect_requested":[]}
72272025-03-02T05:09:00.530ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
72282025-03-02T05:09:00.530ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72292025-03-02T05:09:00.531ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72302025-03-02T05:09:00.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72312025-03-02T05:09:00.534ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
72322025-03-02T05:09:00.534ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
72332025-03-02T05:09:00.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
72342025-03-02T05:09:00.539ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
72352025-03-02T05:09:00.539ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
72362025-03-02T05:09:00.539ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72372025-03-02T05:09:00.539ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 94.813035ms iteration = 14 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
72382025-03-02T05:09:00.540ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 95.755727ms iteration = 79 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
72392025-03-02T05:09:00.541ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72402025-03-02T05:09:00.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72412025-03-02T05:09:00.542ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
72422025-03-02T05:09:00.542ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 0 version = 2
72432025-03-02T05:09:00.542ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
72442025-03-02T05:09:00.542ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
72452025-03-02T05:09:00.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
72462025-03-02T05:09:00.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
72472025-03-02T05:09:00.543ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
72482025-03-02T05:09:00.543ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
72492025-03-02T05:09:00.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
72502025-03-02T05:09:00.543ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72512025-03-02T05:09:00.545ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72522025-03-02T05:09:00.545ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
72532025-03-02T05:09:00.545ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
72542025-03-02T05:09:00.546ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3869, tv_nsec: 773727223 } })
72552025-03-02T05:09:00.546ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
72562025-03-02T05:09:00.548ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
72572025-03-02T05:09:00.548ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
72582025-03-02T05:09:00.548ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
72592025-03-02T05:09:00.548ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
72602025-03-02T05:09:00.548ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
72612025-03-02T05:09:00.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
72622025-03-02T05:09:00.551ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72632025-03-02T05:09:00.553ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 108.061297ms iteration = 27 status = {"errors":[],"requests_created_ok":[]}
72642025-03-02T05:09:00.554ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
72652025-03-02T05:09:00.554ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
72662025-03-02T05:09:00.554ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
72672025-03-02T05:09:00.554ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 01e4ff22-4218-4ece-9bef-81d014113c14 dns_zone_name = oxide.test found_so_far = 1 version = 2
72682025-03-02T05:09:00.554ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
72692025-03-02T05:09:00.554ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-02 04:56:26.100923 UTC dns_group = external generation = 2
72702025-03-02T05:09:00.554ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 115.378882ms iteration = 14 status = {"generation":2}
72712025-03-02T05:09:00.554ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
72722025-03-02T05:09:00.554ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
72732025-03-02T05:09:00.554ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
72742025-03-02T05:09:00.554ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
72752025-03-02T05:09:00.555ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
72762025-03-02T05:09:00.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
72772025-03-02T05:09:00.556ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
72782025-03-02T05:09:00.556ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 112.189641ms iteration = 27 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
72792025-03-02T05:09:00.556ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
72802025-03-02T05:09:00.556ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 115.795868ms iteration = 79 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
72812025-03-02T05:09:00.557ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
72822025-03-02T05:09:00.557ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
72832025-03-02T05:09:00.557ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72842025-03-02T05:09:00.557ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cc5eb28-1bfd-4e43-89f5-ca643056eaee", "content-length": "26", "date": "Sun, 02 Mar 2025 05:09:00 GMT"} })
72852025-03-02T05:09:00.557ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
72862025-03-02T05:09:00.559ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
72872025-03-02T05:09:00.562ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
72882025-03-02T05:09:00.562ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
72892025-03-02T05:09:00.562ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
72902025-03-02T05:09:00.565ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = c1a1b437-f0c3-4f18-8ab5-6f406a5145e4 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
72912025-03-02T05:09:00.565ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
72922025-03-02T05:09:00.565ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-02 04:56:25.487890 UTC dns_group = internal generation = 1
72932025-03-02T05:09:00.565ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 125.974511ms iteration = 14 status = {"generation":1}
72942025-03-02T05:09:00.566ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
72952025-03-02T05:09:00.566ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
72962025-03-02T05:09:00.573ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
72972025-03-02T05:09:00.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
72982025-03-02T05:09:00.585ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
72992025-03-02T05:09:00.585ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
73002025-03-02T05:09:00.585ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 144.087756ms iteration = 14 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
73012025-03-02T05:09:00.595ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 154.464412ms iteration = 29 status = {"physical_disks_added":0}
73022025-03-02T05:09:00.600ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
73032025-03-02T05:09:00.600ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
73042025-03-02T05:09:00.600ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
73052025-03-02T05:09:00.601ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73062025-03-02T05:09:00.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
73072025-03-02T05:09:00.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73082025-03-02T05:09:00.602ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73092025-03-02T05:09:00.603ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 159.004531ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
73102025-03-02T05:09:00.604ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
73112025-03-02T05:09:00.604ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 163.270927ms iteration = 27 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
73122025-03-02T05:09:00.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
73132025-03-02T05:09:00.605ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
73142025-03-02T05:09:00.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
73152025-03-02T05:09:00.605ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 161.172829ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
73162025-03-02T05:09:00.608ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 163.812593ms iteration = 27 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
73172025-03-02T05:09:00.619ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
73182025-03-02T05:09:00.624ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
73192025-03-02T05:09:00.692ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
73202025-03-02T05:09:00.692ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
73212025-03-02T05:09:00.693ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b563b07-4338-434d-8cb6-01a6d3cb90f7", "content-length": "26", "date": "Sun, 02 Mar 2025 05:09:00 GMT"} })
73222025-03-02T05:09:00.693ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
73232025-03-02T05:09:00.695ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
73242025-03-02T05:09:00.703ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
73252025-03-02T05:09:00.703ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
73262025-03-02T05:09:00.703ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8123c17b-d841-41a5-b52c-b05e8c5b0b97", "content-length": "26", "date": "Sun, 02 Mar 2025 05:09:00 GMT"} })
73272025-03-02T05:09:00.703ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
73282025-03-02T05:09:00.705ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
73292025-03-02T05:09:00.885ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
73302025-03-02T05:09:00.885ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
73312025-03-02T05:09:00.885ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3dee650e-81d7-4b25-8408-eae1049813d0", "content-length": "372", "date": "Sun, 02 Mar 2025 05:09:00 GMT"} })
73322025-03-02T05:09:00.885ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73332025-03-02T05:09:00.886ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73342025-03-02T05:09:00.887ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73352025-03-02T05:09:00.889ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 448.613234ms iteration = 27 status = 0
73362025-03-02T05:09:00.890ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
73372025-03-02T05:09:00.890ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
73382025-03-02T05:09:00.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
73392025-03-02T05:09:00.890ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
73402025-03-02T05:09:00.891ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "730ed80b-32f4-4e67-bdb6-3572f68b7d11", "content-length": "2", "date": "Sun, 02 Mar 2025 05:08:59 GMT"} })
73412025-03-02T05:09:00.893ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
73422025-03-02T05:09:00.893ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
73432025-03-02T05:09:00.899ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 460.942604ms iteration = 27 status = null
73442025-03-02T05:09:00.908ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
73452025-03-02T05:09:00.915ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
73462025-03-02T05:09:00.916ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "b03581d8-8c6c-4190-8972-4d1f4d4ae6bc", "date": "Sun, 02 Mar 2025 05:09:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
73472025-03-02T05:09:00.916ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
73482025-03-02T05:09:00.918ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "0b958d04-5391-46bd-86e8-a0a7b57afa0c", "date": "Sun, 02 Mar 2025 05:09:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
73492025-03-02T05:09:00.918ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
73502025-03-02T05:09:00.919ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
73512025-03-02T05:09:00.919ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 480.865353ms iteration = 16 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
73522025-03-02T05:09:00.921ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
73532025-03-02T05:09:00.921ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
73542025-03-02T05:09:00.921ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 477.061934ms iteration = 27 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
73552025-03-02T05:09:00.927ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73562025-03-02T05:09:00.927ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-02T04:55:57.280988Z, time_modified: 2025-03-02T04:55:57.280988Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: b86f11db-859b-43e9-a644-30eba6b1cbfb, db_silo: Silo { identity: SiloIdentity { id: b86f11db-859b-43e9-a644-30eba6b1cbfb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-02T04:56:26.060800Z, time_modified: 2025-03-02T04:56:26.060800Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b" }] }) }
73572025-03-02T05:09:00.927ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 488.058647ms iteration = 14 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]}},"default_endpoint":{"silo_id":"b86f11db-859b-43e9-a644-30eba6b1cbfb","tls_certs":["8cf4239b8d52fb91c26eb1c4d55b1bfce2ddbd99b9909d16153fff44df2f6f2b"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
73582025-03-02T05:09:00.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73592025-03-02T05:09:00.928ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73602025-03-02T05:09:00.930ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
73612025-03-02T05:09:00.930ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73622025-03-02T05:09:00.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73632025-03-02T05:09:00.934ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
73642025-03-02T05:09:00.934ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
73652025-03-02T05:09:00.934ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "43e76bb3-6d63-4004-9198-cc8beb2c637b", "date": "Sun, 02 Mar 2025 05:09:00 GMT"} })
73662025-03-02T05:09:00.934ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73672025-03-02T05:09:00.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73682025-03-02T05:09:00.936ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73692025-03-02T05:09:00.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73702025-03-02T05:09:00.939ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73712025-03-02T05:09:00.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73722025-03-02T05:09:00.942ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
73732025-03-02T05:09:00.942ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
73742025-03-02T05:09:00.943ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "3851f072-e55e-46e6-97ee-a17ac7864f6b", "date": "Sun, 02 Mar 2025 05:09:00 GMT"} })
73752025-03-02T05:09:00.945ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 500.803913ms iteration = 27 status = {}
73762025-03-02T05:09:00.954ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 515.072549ms iteration = 27 status = {}
73772025-03-02T05:09:00.960ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73782025-03-02T05:09:00.961ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
73792025-03-02T05:09:00.961ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73802025-03-02T05:09:00.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73812025-03-02T05:09:00.984ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
73822025-03-02T05:09:00.984ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 540.244247ms iteration = 14 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
73832025-03-02T05:09:00.987ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
73842025-03-02T05:09:00.988ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "a5dbcd86-306a-42c2-b97d-7b1b70133aa8", "date": "Sun, 02 Mar 2025 05:09:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
73852025-03-02T05:09:00.988ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
73862025-03-02T05:09:00.989ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "86a72d0b-090c-4554-8c5b-cb5f304d71dc", "date": "Sun, 02 Mar 2025 05:09:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
73872025-03-02T05:09:00.989ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
73882025-03-02T05:09:00.991ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "006d5e28-a1e3-4e2a-ac0f-6ad0a7083d15", "date": "Sun, 02 Mar 2025 05:09:00 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
73892025-03-02T05:09:00.991ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
73902025-03-02T05:09:00.992ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
73912025-03-02T05:09:00.992ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 553.431809ms iteration = 16 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
73922025-03-02T05:09:01.044ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73932025-03-02T05:09:01.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73942025-03-02T05:09:01.047ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
73952025-03-02T05:09:01.047ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73962025-03-02T05:09:01.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73972025-03-02T05:09:01.057ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
73982025-03-02T05:09:01.057ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "94cb515c-b4d2-4750-b4d9-5b3b5dabb179", "content-length": "134", "date": "Sun, 02 Mar 2025 05:09:00 GMT"} })
73992025-03-02T05:09:01.057ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
74002025-03-02T05:09:01.057ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
74012025-03-02T05:09:01.057ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
74022025-03-02T05:09:01.057ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
74032025-03-02T05:09:01.057ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74042025-03-02T05:09:01.057ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74052025-03-02T05:09:01.057ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74062025-03-02T05:09:01.057ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
74072025-03-02T05:09:01.058ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74082025-03-02T05:09:01.058ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
74092025-03-02T05:09:01.058ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
74102025-03-02T05:09:01.058ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74112025-03-02T05:09:01.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74122025-03-02T05:09:01.060ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
74132025-03-02T05:09:01.060ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "923b58b4-1e0a-45be-9c67-9e5a57305859", "content-length": "2", "date": "Sun, 02 Mar 2025 05:09:00 GMT"} })
74142025-03-02T05:09:01.060ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
74152025-03-02T05:09:01.060ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "672e2496-b287-473d-9a35-d3c2e3287adb", "content-length": "52", "date": "Sun, 02 Mar 2025 05:09:00 GMT"} })
74162025-03-02T05:09:01.060ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
74172025-03-02T05:09:01.060ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
74182025-03-02T05:09:01.061ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
74192025-03-02T05:09:01.153ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "826ffd42-34de-493d-b74a-f062c962e382", "date": "Sun, 02 Mar 2025 05:09:00 GMT"} })
74202025-03-02T05:09:01.153ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
74212025-03-02T05:09:01.153ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
74222025-03-02T05:09:01.153ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c18e1c00-a6c1-4569-a53a-fc6f7db6402f", "content-length": "682", "date": "Sun, 02 Mar 2025 05:09:01 GMT"} })
74232025-03-02T05:09:01.153ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74242025-03-02T05:09:01.153ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74252025-03-02T05:09:01.156ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "090275ca-68e6-42f9-b2ef-091df9c2833d", "content-length": "691", "date": "Sun, 02 Mar 2025 05:09:01 GMT"} })
74262025-03-02T05:09:01.156ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
74272025-03-02T05:09:01.157ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
74282025-03-02T05:09:01.157ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
74292025-03-02T05:09:01.157ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74302025-03-02T05:09:01.157ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74312025-03-02T05:09:01.158ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 714.068751ms iteration = 27 status = {}
74322025-03-02T05:09:01.160ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74332025-03-02T05:09:01.160ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74342025-03-02T05:09:01.162ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74352025-03-02T05:09:01.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74362025-03-02T05:09:01.170ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 726.338441ms iteration = 27 status = {}
74372025-03-02T05:09:02.904ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 311 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 434c57e5-d95d-4fdf-afeb-03efb57ade6e response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
74382025-03-02T05:09:10.441ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 80 reason = Timeout
74392025-03-02T05:09:10.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74402025-03-02T05:09:10.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74412025-03-02T05:09:10.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74422025-03-02T05:09:10.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 80 reason = Timeout
74432025-03-02T05:09:10.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74442025-03-02T05:09:10.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74452025-03-02T05:09:10.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
74462025-03-02T05:09:10.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74472025-03-02T05:09:10.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74482025-03-02T05:09:10.446ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74492025-03-02T05:09:10.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74502025-03-02T05:09:10.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74512025-03-02T05:09:10.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74522025-03-02T05:09:10.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74532025-03-02T05:09:10.452ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
74542025-03-02T05:09:10.452ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74552025-03-02T05:09:10.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74562025-03-02T05:09:10.463ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
74572025-03-02T05:09:10.463ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.897416ms iteration = 80 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
74582025-03-02T05:09:10.476ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.825144ms iteration = 80 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
74592025-03-02T05:09:12.904ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 317 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 29305270-21af-4e8b-840a-4638d0b497e8 response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
74602025-03-02T05:09:20.441ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 81 reason = Timeout
74612025-03-02T05:09:20.441ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74622025-03-02T05:09:20.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74632025-03-02T05:09:20.443ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74642025-03-02T05:09:20.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 81 reason = Timeout
74652025-03-02T05:09:20.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74662025-03-02T05:09:20.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74672025-03-02T05:09:20.445ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
74682025-03-02T05:09:20.446ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74692025-03-02T05:09:20.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74702025-03-02T05:09:20.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74712025-03-02T05:09:20.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74722025-03-02T05:09:20.449ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74732025-03-02T05:09:20.451ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
74742025-03-02T05:09:20.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74752025-03-02T05:09:20.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74762025-03-02T05:09:20.461ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
74772025-03-02T05:09:20.461ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.709987ms iteration = 81 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
74782025-03-02T05:09:20.474ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74792025-03-02T05:09:20.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74802025-03-02T05:09:20.476ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.775483ms iteration = 81 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
74812025-03-02T05:09:22.904ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 290 local_addr = [fd00:1122:3344:101::b]:59682 method = GET remote_addr = [fd00:1122:3344:101::d]:54673 req_id = 4964bed5-88b7-4ff4-b691-cebd67cc9c7a response_code = 200 uri = /aebed526-5e03-4a58-86b5-1baf17c88392
74822025-03-02T05:09:27.897ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:47091
74832025-03-02T05:09:27.897ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47091 req_id = 0411fd2b-7131-4cbc-9da7-bbfe28b43053 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
74842025-03-02T05:09:27.897ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47091 req_id = 0411fd2b-7131-4cbc-9da7-bbfe28b43053 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
74852025-03-02T05:09:27.900ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2869 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47091 req_id = 0411fd2b-7131-4cbc-9da7-bbfe28b43053 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?sort_by=id_ascending
74862025-03-02T05:09:27.900ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47091 req_id = 045820ed-497b-4eec-a082-f0acb6be4975 roles = RoleSet { roles: {} } uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
74872025-03-02T05:09:27.900ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47091 req_id = 045820ed-497b-4eec-a082-f0acb6be4975 resource = Database result = Ok(()) uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
74882025-03-02T05:09:27.903ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2561 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47091 req_id = 045820ed-497b-4eec-a082-f0acb6be4975 response_code = 200 uri = /metrics/collectors/8aaa536c-c534-4dd2-aad0-92d8934e5eba/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmZWUxN2M0NS1hODViLTQwNWYtOWU1OC0yY2EwYWIwM2RlYTkifX0%3D
74892025-03-02T05:09:30.439ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 28 reason = Timeout
74902025-03-02T05:09:30.439ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 28 reason = Timeout
74912025-03-02T05:09:30.439ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 28 reason = Timeout
74922025-03-02T05:09:30.439ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
74932025-03-02T05:09:30.439ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
74942025-03-02T05:09:30.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
74952025-03-02T05:09:30.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
74962025-03-02T05:09:30.442ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 82 reason = Timeout
74972025-03-02T05:09:30.442ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 28 reason = Timeout
74982025-03-02T05:09:30.442ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 28 reason = Timeout
74992025-03-02T05:09:30.442ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
75002025-03-02T05:09:30.442ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75012025-03-02T05:09:30.442ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 30 reason = Timeout
75022025-03-02T05:09:30.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75032025-03-02T05:09:30.442ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
75042025-03-02T05:09:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
75052025-03-02T05:09:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75062025-03-02T05:09:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75072025-03-02T05:09:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
75082025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 28 reason = Timeout
75092025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
75102025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 28 reason = Timeout
75112025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 28 reason = Timeout
75122025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
75132025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
75142025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 28 reason = Timeout
75152025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
75162025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
75172025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 28 reason = Timeout
75182025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
75192025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
75202025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 28 reason = Timeout
75212025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
75222025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
75232025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 28 reason = Timeout
75242025-03-02T05:09:30.444ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
75252025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 28 reason = Timeout
75262025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
75272025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 28 reason = Timeout
75282025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 82 reason = Timeout
75292025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
75302025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
75312025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
75322025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75332025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 28 reason = Timeout
75342025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
75352025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
75362025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
75372025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
75382025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75392025-03-02T05:09:30.444ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75402025-03-02T05:09:30.445ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3899, tv_nsec: 672504338 } })
75412025-03-02T05:09:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
75422025-03-02T05:09:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
75432025-03-02T05:09:30.445ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3899, tv_nsec: 672707951 } })
75442025-03-02T05:09:30.445ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 28 reason = Timeout
75452025-03-02T05:09:30.445ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
75462025-03-02T05:09:30.445ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
75472025-03-02T05:09:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
75482025-03-02T05:09:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75492025-03-02T05:09:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
75502025-03-02T05:09:30.445ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
75512025-03-02T05:09:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
75522025-03-02T05:09:30.447ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75532025-03-02T05:09:30.447ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75542025-03-02T05:09:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
75552025-03-02T05:09:30.448ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
75562025-03-02T05:09:30.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
75572025-03-02T05:09:30.448ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 9.678866ms iteration = 28 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
75582025-03-02T05:09:30.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
75592025-03-02T05:09:30.449ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75602025-03-02T05:09:30.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75612025-03-02T05:09:30.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
75622025-03-02T05:09:30.449ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
75632025-03-02T05:09:30.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
75642025-03-02T05:09:30.451ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
75652025-03-02T05:09:30.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75662025-03-02T05:09:30.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75672025-03-02T05:09:30.453ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75682025-03-02T05:09:30.455ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
75692025-03-02T05:09:30.455ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
75702025-03-02T05:09:30.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
75712025-03-02T05:09:30.457ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75722025-03-02T05:09:30.459ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
75732025-03-02T05:09:30.459ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75742025-03-02T05:09:30.459ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75752025-03-02T05:09:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75762025-03-02T05:09:30.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75772025-03-02T05:09:30.468ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75782025-03-02T05:09:30.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75792025-03-02T05:09:30.470ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
75802025-03-02T05:09:30.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
75812025-03-02T05:09:30.471ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75822025-03-02T05:09:30.471ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-fee17c45-a85b-405f-9e58-2ca0ab03dea9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3899, tv_nsec: 699186246 } })
75832025-03-02T05:09:30.472ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
75842025-03-02T05:09:30.472ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
75852025-03-02T05:09:30.472ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75862025-03-02T05:09:30.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75872025-03-02T05:09:30.502ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75882025-03-02T05:09:30.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75892025-03-02T05:09:30.506ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 62.526325ms iteration = 28 status = {"errors":[],"finish_invoked_ok":[]}
75902025-03-02T05:09:30.507ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
75912025-03-02T05:09:30.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
75922025-03-02T05:09:30.509ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
75932025-03-02T05:09:30.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
75942025-03-02T05:09:30.510ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
75952025-03-02T05:09:30.510ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
75962025-03-02T05:09:30.510ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
75972025-03-02T05:09:30.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
75982025-03-02T05:09:30.511ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75992025-03-02T05:09:30.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76002025-03-02T05:09:30.516ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
76012025-03-02T05:09:30.516ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
76022025-03-02T05:09:30.516ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
76032025-03-02T05:09:30.516ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
76042025-03-02T05:09:30.517ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76052025-03-02T05:09:30.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
76062025-03-02T05:09:30.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
76072025-03-02T05:09:30.518ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 74.135455ms iteration = 28 status = {"errors":[],"garbage_collect_requested":[]}
76082025-03-02T05:09:30.520ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6231c748-6e83-4b98-9086-de7a75b0ccca, lookup_type: ById(6231c748-6e83-4b98-9086-de7a75b0ccca) } result = Ok(())
76092025-03-02T05:09:30.520ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76102025-03-02T05:09:30.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76112025-03-02T05:09:30.522ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 77.931874ms iteration = 28 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
76122025-03-02T05:09:30.522ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
76132025-03-02T05:09:30.522ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
76142025-03-02T05:09:30.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
76152025-03-02T05:09:30.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
76162025-03-02T05:09:30.524ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
76172025-03-02T05:09:30.524ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
76182025-03-02T05:09:30.524ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76192025-03-02T05:09:30.524ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "399664d1-4154-4351-9f1e-5a6f01d5ea07", "content-length": "26", "date": "Sun, 02 Mar 2025 05:09:30 GMT"} })
76202025-03-02T05:09:30.524ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
76212025-03-02T05:09:30.526ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
76222025-03-02T05:09:30.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
76232025-03-02T05:09:30.526ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76242025-03-02T05:09:30.526ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
76252025-03-02T05:09:30.526ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
76262025-03-02T05:09:30.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
76272025-03-02T05:09:30.527ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
76282025-03-02T05:09:30.529ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 84.161035ms iteration = 28 status = {"errors":[],"requests_created_ok":[]}
76292025-03-02T05:09:30.529ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
76302025-03-02T05:09:30.529ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
76312025-03-02T05:09:30.530ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
76322025-03-02T05:09:30.530ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76332025-03-02T05:09:30.533ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
76342025-03-02T05:09:30.533ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
76352025-03-02T05:09:30.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
76362025-03-02T05:09:30.545ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca original_time_created = 2025-03-02 04:55:50.528243 UTC target_id = 6231c748-6e83-4b98-9086-de7a75b0ccca time_created = 2025-03-02 04:55:50.528243 UTC
76372025-03-02T05:09:30.545ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 103.095512ms iteration = 82 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"6231c748-6e83-4b98-9086-de7a75b0ccca","time_created":"2025-03-02T04:55:50.528243Z"}
76382025-03-02T05:09:30.550ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
76392025-03-02T05:09:30.550ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
76402025-03-02T05:09:30.551ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "356196e0-65b9-4658-97e1-c69e50c5ea61", "content-length": "26", "date": "Sun, 02 Mar 2025 05:09:30 GMT"} })
76412025-03-02T05:09:30.551ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
76422025-03-02T05:09:30.552ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
76432025-03-02T05:09:30.563ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 118.953948ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
76442025-03-02T05:09:30.567ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
76452025-03-02T05:09:30.567ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
76462025-03-02T05:09:30.567ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 122.823279ms iteration = 28 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
76472025-03-02T05:09:30.568ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 124.543911ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
76482025-03-02T05:09:30.569ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76492025-03-02T05:09:30.573ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: fee17c45-a85b-405f-9e58-2ca0ab03dea9, lookup_type: ById(fee17c45-a85b-405f-9e58-2ca0ab03dea9) } result = Ok(())
76502025-03-02T05:09:30.573ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76512025-03-02T05:09:30.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76522025-03-02T05:09:30.575ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 133.490508ms iteration = 28 status = 0
76532025-03-02T05:09:30.577ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
76542025-03-02T05:09:30.577ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 135.576815ms iteration = 28 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
76552025-03-02T05:09:30.582ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 138.167619ms iteration = 82 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
76562025-03-02T05:09:30.592ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
76572025-03-02T05:09:30.592ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 147.385109ms iteration = 28 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
76582025-03-02T05:09:30.636ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
76592025-03-02T05:09:30.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
76602025-03-02T05:09:30.638ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = d6094382-b327-4998-b53d-b2d98a53005b zone_address = fd00:1122:3344:101::2
76612025-03-02T05:09:30.638ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
76622025-03-02T05:09:30.639ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f00f1864-a34f-452e-80a8-b7098a3aaf8f", "content-length": "26", "date": "Sun, 02 Mar 2025 05:09:30 GMT"} })
76632025-03-02T05:09:30.639ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = d6094382-b327-4998-b53d-b2d98a53005b slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
76642025-03-02T05:09:30.640ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = d6094382-b327-4998-b53d-b2d98a53005b
76652025-03-02T05:09:30.654ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 211.96605ms iteration = 30 status = {"physical_disks_added":0}
76662025-03-02T05:09:30.683ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
76672025-03-02T05:09:30.684ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "98310686-a22a-46b8-b622-656f9fb12377", "content-length": "2", "date": "Sun, 02 Mar 2025 05:09:30 GMT"} })
76682025-03-02T05:09:30.686ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
76692025-03-02T05:09:30.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
76702025-03-02T05:09:30.687ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 248.225313ms iteration = 28 status = {}
76712025-03-02T05:09:30.690ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
76722025-03-02T05:09:30.690ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
76732025-03-02T05:09:30.690ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3580eb5e-c77e-42f2-ba96-637711430447", "content-length": "372", "date": "Sun, 02 Mar 2025 05:09:30 GMT"} })
76742025-03-02T05:09:30.690ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
76752025-03-02T05:09:30.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
76762025-03-02T05:09:30.693ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 254.031048ms iteration = 28 status = null
76772025-03-02T05:09:30.728ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
76782025-03-02T05:09:30.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
76792025-03-02T05:09:30.740ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
76802025-03-02T05:09:30.740ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
76812025-03-02T05:09:30.740ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "d2f33404-df11-4932-a076-068a52b30378", "date": "Sun, 02 Mar 2025 05:09:30 GMT"} })
76822025-03-02T05:09:30.740ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
76832025-03-02T05:09:30.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
76842025-03-02T05:09:30.742ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
76852025-03-02T05:09:30.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
76862025-03-02T05:09:30.745ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
76872025-03-02T05:09:30.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
76882025-03-02T05:09:30.748ZINFOnexus (BackgroundTasks): VPC route manager sled fee17c45-a85b-405f-9e58-2ca0ab03dea9 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
76892025-03-02T05:09:30.748ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
76902025-03-02T05:09:30.761ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76912025-03-02T05:09:30.763ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
76922025-03-02T05:09:30.763ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
76932025-03-02T05:09:30.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
76942025-03-02T05:09:30.764ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76952025-03-02T05:09:30.766ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
76962025-03-02T05:09:30.766ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
76972025-03-02T05:09:30.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
76982025-03-02T05:09:30.776ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
76992025-03-02T05:09:30.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77002025-03-02T05:09:30.778ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 905d8919-6499-4405-90a1-a14b31c7fd5e, rack_id: d6094382-b327-4998-b53d-b2d98a53005b, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6c936df1-20e2-41ff-9a89-c03d8807ba9b) } rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
77012025-03-02T05:09:30.778ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77022025-03-02T05:09:30.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77032025-03-02T05:09:30.788ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
77042025-03-02T05:09:30.788ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "5529ea09-90f0-473c-8cf8-0933bc0506cb", "content-length": "134", "date": "Sun, 02 Mar 2025 05:09:30 GMT"} })
77052025-03-02T05:09:30.788ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = d6094382-b327-4998-b53d-b2d98a53005b settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
77062025-03-02T05:09:30.788ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6c936df1-20e2-41ff-9a89-c03d8807ba9b
77072025-03-02T05:09:30.789ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
77082025-03-02T05:09:30.789ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
77092025-03-02T05:09:30.789ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77102025-03-02T05:09:30.789ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77112025-03-02T05:09:30.789ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77122025-03-02T05:09:30.789ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
77132025-03-02T05:09:30.789ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = d6094382-b327-4998-b53d-b2d98a53005b routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
77142025-03-02T05:09:30.789ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = d6094382-b327-4998-b53d-b2d98a53005b request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
77152025-03-02T05:09:30.789ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = d6094382-b327-4998-b53d-b2d98a53005b
77162025-03-02T05:09:30.789ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77172025-03-02T05:09:30.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77182025-03-02T05:09:30.791ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
77192025-03-02T05:09:30.791ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9b66e1b-fb3a-409c-a3a2-3b5d6dc6e967", "content-length": "2", "date": "Sun, 02 Mar 2025 05:09:30 GMT"} })
77202025-03-02T05:09:30.791ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
77212025-03-02T05:09:30.791ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cc0860d-64bb-48ad-8bec-45f56bd23fa0", "content-length": "52", "date": "Sun, 02 Mar 2025 05:09:30 GMT"} })
77222025-03-02T05:09:30.791ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch_location = Switch0
77232025-03-02T05:09:30.791ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
77242025-03-02T05:09:30.850ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "f4b83574-db69-477d-95ab-ca2622bc80a0", "date": "Sun, 02 Mar 2025 05:09:30 GMT"} })
77252025-03-02T05:09:30.852ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 407.397606ms iteration = 28 status = {}
77262025-03-02T05:09:30.864ZDEBGnexus (BackgroundTasks): client request SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
77272025-03-02T05:09:30.869ZDEBGnexus (BackgroundTasks): client response SledAgent = fee17c45-a85b-405f-9e58-2ca0ab03dea9 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5cee107-3e2a-4c66-a4c8-6ad0d2432fd8", "content-length": "691", "date": "Sun, 02 Mar 2025 05:09:30 GMT"} })
77282025-03-02T05:09:30.869ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
77292025-03-02T05:09:30.869ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
77302025-03-02T05:09:30.869ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
77312025-03-02T05:09:30.870ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 426.09565ms iteration = 28 status = {}
77322025-03-02T05:09:30.940ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "145061cd-1986-4edd-af0c-39582b59deb8", "date": "Sun, 02 Mar 2025 05:09:30 GMT"} })
77332025-03-02T05:09:30.940ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = d6094382-b327-4998-b53d-b2d98a53005b switch = Switch0
77342025-03-02T05:09:30.940ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = d6094382-b327-4998-b53d-b2d98a53005b uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
77352025-03-02T05:09:30.940ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = d6094382-b327-4998-b53d-b2d98a53005b result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "172cfa4a-7fa8-475b-9c62-baffa7fa1247", "content-length": "682", "date": "Sun, 02 Mar 2025 05:09:30 GMT"} })
77362025-03-02T05:09:30.940ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77372025-03-02T05:09:30.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77382025-03-02T05:09:30.944ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77392025-03-02T05:09:30.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77402025-03-02T05:09:30.947ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77412025-03-02T05:09:30.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77422025-03-02T05:09:30.949ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77432025-03-02T05:09:30.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77442025-03-02T05:09:30.957ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 512.83118ms iteration = 28 status = {}