Max level shown:
12024-02-15T07:00:09.357ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 6 reason = Timeout
22024-02-15T07:00:09.357ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
32024-02-15T07:00:09.357ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 6 reason = Timeout
42024-02-15T07:00:09.357ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
52024-02-15T07:00:09.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
62024-02-15T07:00:09.358ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("78a9790e-f724-40b6-bd7f-e718315a779a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("de8bbd15-7094-44ed-a766-1edae8657a5e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ffdb0d80-1b9f-4db4-a0b6-96d3f7810c28.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ffdb0d80-1b9f-4db4-a0b6-96d3f7810c28.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 1836, tv_nsec: 184142269 } })
72024-02-15T07:00:09.358ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
82024-02-15T07:00:09.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
92024-02-15T07:00:09.360ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102024-02-15T07:00:09.363ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
112024-02-15T07:00:09.363ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 6.209163ms iteration = 6
122024-02-15T07:00:09.364ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
132024-02-15T07:00:09.364ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 10 reason = Timeout
142024-02-15T07:00:09.364ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
152024-02-15T07:00:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 6 reason = Timeout
162024-02-15T07:00:09.364ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
172024-02-15T07:00:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 6 reason = Timeout
182024-02-15T07:00:09.364ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
192024-02-15T07:00:09.364ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 6 reason = Timeout
202024-02-15T07:00:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 8 reason = Timeout
212024-02-15T07:00:09.364ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
222024-02-15T07:00:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 8 reason = Timeout
232024-02-15T07:00:09.364ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 10 reason = Timeout
242024-02-15T07:00:09.364ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
252024-02-15T07:00:09.365ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
262024-02-15T07:00:09.365ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
272024-02-15T07:00:09.365ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
282024-02-15T07:00:09.365ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
292024-02-15T07:00:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
302024-02-15T07:00:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
312024-02-15T07:00:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
322024-02-15T07:00:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
332024-02-15T07:00:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
342024-02-15T07:00:09.365ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
352024-02-15T07:00:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
362024-02-15T07:00:09.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 26 reason = Timeout
372024-02-15T07:00:09.366ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 10 reason = Timeout
382024-02-15T07:00:09.366ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
392024-02-15T07:00:09.366ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 6 reason = Timeout
402024-02-15T07:00:09.366ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:46 reason = no blueprint
412024-02-15T07:00:09.366ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
422024-02-15T07:00:09.366ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 7.54\u{b5}s iteration = 6
432024-02-15T07:00:09.366ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 10 reason = Timeout
442024-02-15T07:00:09.366ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
452024-02-15T07:00:09.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
462024-02-15T07:00:09.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
472024-02-15T07:00:09.366ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
482024-02-15T07:00:09.367ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
492024-02-15T07:00:09.368ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
502024-02-15T07:00:09.368ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
512024-02-15T07:00:09.368ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
522024-02-15T07:00:09.368ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.951493ms iteration = 10
532024-02-15T07:00:09.369ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
542024-02-15T07:00:09.369ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("19feddd6-78c6-4e31-b93a-acace968955e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("32bd8fe1-df5d-4887-b6be-0cddecf4285c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("32bd8fe1-df5d-4887-b6be-0cddecf4285c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::9)) }], valid_until: Instant { tv_sec: 1836, tv_nsec: 195424322 } })
552024-02-15T07:00:09.369ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
562024-02-15T07:00:09.370ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
572024-02-15T07:00:09.370ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "92a9cc83-5110-42a8-83a8-ed5174c3106f", "content-length": "2", "date": "Thu, 15 Feb 2024 07:00:09 GMT"} })
582024-02-15T07:00:09.370ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 6.107191ms iteration = 10
592024-02-15T07:00:09.371ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "0cb66547-5637-48d9-a609-6a57d725276e", "content-length": "2", "date": "Thu, 15 Feb 2024 07:00:08 GMT"} })
602024-02-15T07:00:09.371ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
612024-02-15T07:00:09.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
622024-02-15T07:00:09.372ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
632024-02-15T07:00:09.372ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
642024-02-15T07:00:09.372ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 7.805278ms iteration = 6
652024-02-15T07:00:09.373ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
662024-02-15T07:00:09.373ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
672024-02-15T07:00:09.373ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
682024-02-15T07:00:09.373ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
692024-02-15T07:00:09.373ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
702024-02-15T07:00:09.373ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
712024-02-15T07:00:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
722024-02-15T07:00:09.373ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
732024-02-15T07:00:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
742024-02-15T07:00:09.373ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 8.597225ms iteration = 10
752024-02-15T07:00:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
762024-02-15T07:00:09.373ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
772024-02-15T07:00:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
782024-02-15T07:00:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
792024-02-15T07:00:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
802024-02-15T07:00:09.374ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 10 reason = Timeout
812024-02-15T07:00:09.374ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
822024-02-15T07:00:09.374ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
832024-02-15T07:00:09.374ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 11.54\u{b5}s iteration = 10
842024-02-15T07:00:09.375ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 9.069495ms iteration = 26
852024-02-15T07:00:09.375ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
862024-02-15T07:00:09.375ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = d7f8cbe9-558e-41ea-b105-dcc67f70cc11 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
872024-02-15T07:00:09.375ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
882024-02-15T07:00:09.375ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
892024-02-15T07:00:09.375ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
902024-02-15T07:00:09.375ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
912024-02-15T07:00:09.375ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
922024-02-15T07:00:09.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
932024-02-15T07:00:09.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
942024-02-15T07:00:09.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
952024-02-15T07:00:09.376ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
962024-02-15T07:00:09.376ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
972024-02-15T07:00:09.376ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
982024-02-15T07:00:09.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
992024-02-15T07:00:09.380ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
1002024-02-15T07:00:09.380ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1012024-02-15T07:00:09.380ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
1022024-02-15T07:00:09.380ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1032024-02-15T07:00:09.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1042024-02-15T07:00:09.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1052024-02-15T07:00:09.380ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
1062024-02-15T07:00:09.380ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1072024-02-15T07:00:09.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1082024-02-15T07:00:09.382ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
1092024-02-15T07:00:09.382ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = c0170a00-b2b4-4879-8162-cfda0a5009a3 dns_zone_name = oxide.test found_so_far = 0 version = 2
1102024-02-15T07:00:09.382ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1112024-02-15T07:00:09.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1122024-02-15T07:00:09.382ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = d7f8cbe9-558e-41ea-b105-dcc67f70cc11 dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
1132024-02-15T07:00:09.382ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
1142024-02-15T07:00:09.382ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-02-15 06:56:35.135865 UTC dns_group = internal generation = 1
1152024-02-15T07:00:09.382ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 24.830594ms iteration = 6
1162024-02-15T07:00:09.382ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1172024-02-15T07:00:09.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1182024-02-15T07:00:09.383ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1192024-02-15T07:00:09.383ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1202024-02-15T07:00:09.386ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
1212024-02-15T07:00:09.386ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1222024-02-15T07:00:09.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1232024-02-15T07:00:09.386ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
1242024-02-15T07:00:09.386ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1252024-02-15T07:00:09.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1262024-02-15T07:00:09.389ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = c0170a00-b2b4-4879-8162-cfda0a5009a3 dns_zone_name = oxide.test found_so_far = 1 version = 2
1272024-02-15T07:00:09.389ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
1282024-02-15T07:00:09.389ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-02-15 06:56:35.417009 UTC dns_group = external generation = 2
1292024-02-15T07:00:09.389ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 24.585958ms iteration = 6
1302024-02-15T07:00:09.397ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1312024-02-15T07:00:09.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1322024-02-15T07:00:09.398ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-02-15T06:56:09.420632Z, time_modified: 2024-02-15T06:56:09.420632Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ffb05f40-4544-4250-8e84-31ffead32a4d, db_silo: Silo { identity: SiloIdentity { id: ffb05f40-4544-4250-8e84-31ffead32a4d, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T06:56:35.393136Z, time_modified: 2024-02-15T06:56:35.393136Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc23126eba6ce80ab4599432cb1e3814af0b1a2121110ea710e2bb25f380b4e9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ffb05f40-4544-4250-8e84-31ffead32a4d, db_silo: Silo { identity: SiloIdentity { id: ffb05f40-4544-4250-8e84-31ffead32a4d, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T06:56:35.393136Z, time_modified: 2024-02-15T06:56:35.393136Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc23126eba6ce80ab4599432cb1e3814af0b1a2121110ea710e2bb25f380b4e9" }] }) }
1332024-02-15T07:00:09.398ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 33.52029ms iteration = 6
1342024-02-15T07:00:09.399ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1352024-02-15T07:00:09.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1362024-02-15T07:00:09.400ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1372024-02-15T07:00:09.403ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ec8d62a8-6b82-4888-b333-60568d39e395, lookup_type: ById(ec8d62a8-6b82-4888-b333-60568d39e395) } result = Ok(())
1382024-02-15T07:00:09.403ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1392024-02-15T07:00:09.403ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1402024-02-15T07:00:09.404ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 38.877795ms iteration = 10
1412024-02-15T07:00:09.430ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
1422024-02-15T07:00:09.433ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
1432024-02-15T07:00:09.452ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = f446856a-64c8-4576-8ca0-a0bb364193a8 roles = RoleSet { roles: {} } uri = /nat/ipv4/changeset/11?limit=100
1442024-02-15T07:00:09.453ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = f446856a-64c8-4576-8ca0-a0bb364193a8 resource = Database result = Ok(()) uri = /nat/ipv4/changeset/11?limit=100
1452024-02-15T07:00:09.455ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 3134 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = f446856a-64c8-4576-8ca0-a0bb364193a8 response_code = 200 uri = /nat/ipv4/changeset/11?limit=100
1462024-02-15T07:00:09.489ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
1472024-02-15T07:00:09.505ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
1482024-02-15T07:00:09.505ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "5bb580db-95cb-4a7d-90ea-8d34500db188", "date": "Thu, 15 Feb 2024 07:00:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
1492024-02-15T07:00:09.505ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
1502024-02-15T07:00:09.507ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "5fa6f64c-b148-4703-960f-5cbbbd905bad", "date": "Thu, 15 Feb 2024 07:00:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
1512024-02-15T07:00:09.507ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
1522024-02-15T07:00:09.508ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
1532024-02-15T07:00:09.508ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 143.658505ms iteration = 8
1542024-02-15T07:00:09.527ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
1552024-02-15T07:00:09.528ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "2dcc33b5-188b-415c-8e12-c1d0f5b91570", "date": "Thu, 15 Feb 2024 07:00:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
1562024-02-15T07:00:09.528ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
1572024-02-15T07:00:09.530ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "e513a971-05bc-41ef-98f7-d3601d2e2087", "date": "Thu, 15 Feb 2024 07:00:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
1582024-02-15T07:00:09.530ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
1592024-02-15T07:00:09.531ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "dc2df329-30d5-4e9c-b321-17d5d20aca7b", "date": "Thu, 15 Feb 2024 07:00:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
1602024-02-15T07:00:09.531ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
1612024-02-15T07:00:09.533ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
1622024-02-15T07:00:09.533ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 168.909927ms iteration = 8
1632024-02-15T07:00:09.684ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 883 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 39a8d978-c0b3-4466-a3af-b5d62a724137 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
1642024-02-15T07:00:19.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 27 reason = Timeout
1652024-02-15T07:00:19.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1662024-02-15T07:00:19.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1672024-02-15T07:00:19.367ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1682024-02-15T07:00:19.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1692024-02-15T07:00:19.370ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1702024-02-15T07:00:19.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1712024-02-15T07:00:19.371ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.807464ms iteration = 27
1722024-02-15T07:00:19.684ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 616 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = f9bee496-a525-4339-aeaf-66dcff084f17 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
1732024-02-15T07:00:29.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 28 reason = Timeout
1742024-02-15T07:00:29.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1752024-02-15T07:00:29.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1762024-02-15T07:00:29.367ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1772024-02-15T07:00:29.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1782024-02-15T07:00:29.370ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1792024-02-15T07:00:29.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1802024-02-15T07:00:29.371ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.879756ms iteration = 28
1812024-02-15T07:00:29.683ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 659 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = a34ddbee-7896-471a-b16f-c033942747cc response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
1822024-02-15T07:00:39.364ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 11 reason = Timeout
1832024-02-15T07:00:39.364ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
1842024-02-15T07:00:39.364ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 11 reason = Timeout
1852024-02-15T07:00:39.364ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
1862024-02-15T07:00:39.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1872024-02-15T07:00:39.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
1882024-02-15T07:00:39.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 29 reason = Timeout
1892024-02-15T07:00:39.365ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 11 reason = Timeout
1902024-02-15T07:00:39.365ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
1912024-02-15T07:00:39.365ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 11 reason = Timeout
1922024-02-15T07:00:39.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1932024-02-15T07:00:39.365ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1942024-02-15T07:00:39.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1952024-02-15T07:00:39.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1962024-02-15T07:00:39.365ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
1972024-02-15T07:00:39.366ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "02c0ee7c-a595-41db-98ce-9cf8f5b31c34", "content-length": "2", "date": "Thu, 15 Feb 2024 07:00:38 GMT"} })
1982024-02-15T07:00:39.366ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
1992024-02-15T07:00:39.366ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
2002024-02-15T07:00:39.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
2012024-02-15T07:00:39.367ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2022024-02-15T07:00:39.367ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "66ed3704-ec8f-4bcb-a147-92a068bea668", "content-length": "2", "date": "Thu, 15 Feb 2024 07:00:39 GMT"} })
2032024-02-15T07:00:39.367ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2042024-02-15T07:00:39.367ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 2.75822ms iteration = 11
2052024-02-15T07:00:39.367ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 3.586187ms iteration = 11
2062024-02-15T07:00:39.368ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
2072024-02-15T07:00:39.368ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.991824ms iteration = 11
2082024-02-15T07:00:39.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
2092024-02-15T07:00:39.370ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2102024-02-15T07:00:39.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2112024-02-15T07:00:39.370ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2122024-02-15T07:00:39.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2132024-02-15T07:00:39.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2142024-02-15T07:00:39.372ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.646262ms iteration = 29
2152024-02-15T07:00:39.372ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2162024-02-15T07:00:39.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2172024-02-15T07:00:39.374ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 11 reason = Timeout
2182024-02-15T07:00:39.374ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
2192024-02-15T07:00:39.374ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
2202024-02-15T07:00:39.374ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 8.93\u{b5}s iteration = 11
2212024-02-15T07:00:39.386ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2222024-02-15T07:00:39.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2232024-02-15T07:00:39.387ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2242024-02-15T07:00:39.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2252024-02-15T07:00:39.388ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2262024-02-15T07:00:39.391ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ec8d62a8-6b82-4888-b333-60568d39e395, lookup_type: ById(ec8d62a8-6b82-4888-b333-60568d39e395) } result = Ok(())
2272024-02-15T07:00:39.391ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2282024-02-15T07:00:39.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2292024-02-15T07:00:39.392ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 27.169584ms iteration = 11
2302024-02-15T07:00:39.485ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 7d0a8761-0781-4223-b09c-d2e5ed239e07 roles = RoleSet { roles: {} } uri = /nat/ipv4/changeset/11?limit=100
2312024-02-15T07:00:39.485ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 7d0a8761-0781-4223-b09c-d2e5ed239e07 resource = Database result = Ok(()) uri = /nat/ipv4/changeset/11?limit=100
2322024-02-15T07:00:39.488ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 3110 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 7d0a8761-0781-4223-b09c-d2e5ed239e07 response_code = 200 uri = /nat/ipv4/changeset/11?limit=100
2332024-02-15T07:00:39.684ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 1179 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 5dff87a1-d5e8-409d-a954-41b259996d0b response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
2342024-02-15T07:00:49.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 30 reason = Timeout
2352024-02-15T07:00:49.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2362024-02-15T07:00:49.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2372024-02-15T07:00:49.367ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2382024-02-15T07:00:49.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2392024-02-15T07:00:49.370ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2402024-02-15T07:00:49.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2412024-02-15T07:00:49.371ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.671512ms iteration = 30
2422024-02-15T07:00:49.684ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 783 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 63cd0658-6177-481e-b11f-3b522f5a1de5 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
2432024-02-15T07:00:59.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 31 reason = Timeout
2442024-02-15T07:00:59.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2452024-02-15T07:00:59.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2462024-02-15T07:00:59.367ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2472024-02-15T07:00:59.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2482024-02-15T07:00:59.370ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2492024-02-15T07:00:59.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2502024-02-15T07:00:59.371ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.937827ms iteration = 31
2512024-02-15T07:00:59.684ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 849 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = a3eb84fa-03b5-474f-ad5a-d07fc3b98024 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
2522024-02-15T07:01:09.357ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 7 reason = Timeout
2532024-02-15T07:01:09.357ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
2542024-02-15T07:01:09.357ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 7 reason = Timeout
2552024-02-15T07:01:09.357ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
2562024-02-15T07:01:09.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
2572024-02-15T07:01:09.358ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
2582024-02-15T07:01:09.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
2592024-02-15T07:01:09.360ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2602024-02-15T07:01:09.361ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("78a9790e-f724-40b6-bd7f-e718315a779a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("de8bbd15-7094-44ed-a766-1edae8657a5e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ffdb0d80-1b9f-4db4-a0b6-96d3f7810c28.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ffdb0d80-1b9f-4db4-a0b6-96d3f7810c28.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 1896, tv_nsec: 187560202 } })
2612024-02-15T07:01:09.363ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
2622024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 12 reason = Timeout
2632024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 7 reason = Timeout
2642024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
2652024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2662024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 7 reason = Timeout
2672024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 7 reason = Timeout
2682024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
2692024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 9 reason = Timeout
2702024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
2712024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 12 reason = Timeout
2722024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
2732024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 9 reason = Timeout
2742024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 32 reason = Timeout
2752024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 12 reason = Timeout
2762024-02-15T07:01:09.364ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
2772024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2782024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 7 reason = Timeout
2792024-02-15T07:01:09.364ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 12 reason = Timeout
2802024-02-15T07:01:09.365ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:46 reason = no blueprint
2812024-02-15T07:01:09.365ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
2822024-02-15T07:01:09.365ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
2832024-02-15T07:01:09.365ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
2842024-02-15T07:01:09.365ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.691\u{b5}s iteration = 7
2852024-02-15T07:01:09.365ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2862024-02-15T07:01:09.365ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
2872024-02-15T07:01:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2882024-02-15T07:01:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
2892024-02-15T07:01:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
2902024-02-15T07:01:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
2912024-02-15T07:01:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2922024-02-15T07:01:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2932024-02-15T07:01:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
2942024-02-15T07:01:09.367ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2952024-02-15T07:01:09.367ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2962024-02-15T07:01:09.367ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
2972024-02-15T07:01:09.368ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
2982024-02-15T07:01:09.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
2992024-02-15T07:01:09.368ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3002024-02-15T07:01:09.368ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3012024-02-15T07:01:09.368ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
3022024-02-15T07:01:09.368ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.596457ms iteration = 12
3032024-02-15T07:01:09.369ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "db563497-fb0b-45b4-8e06-f4cdf3b1cd9b", "content-length": "2", "date": "Thu, 15 Feb 2024 07:01:09 GMT"} })
3042024-02-15T07:01:09.369ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 4.236431ms iteration = 12
3052024-02-15T07:01:09.369ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
3062024-02-15T07:01:09.369ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 11.563818ms iteration = 7
3072024-02-15T07:01:09.369ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
3082024-02-15T07:01:09.369ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("19feddd6-78c6-4e31-b93a-acace968955e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("32bd8fe1-df5d-4887-b6be-0cddecf4285c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("32bd8fe1-df5d-4887-b6be-0cddecf4285c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::9)) }], valid_until: Instant { tv_sec: 1896, tv_nsec: 195459222 } })
3092024-02-15T07:01:09.369ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3102024-02-15T07:01:09.370ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "8b218d6d-a6b7-4e41-84ac-ad4dae45de8a", "content-length": "2", "date": "Thu, 15 Feb 2024 07:01:09 GMT"} })
3112024-02-15T07:01:09.370ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3122024-02-15T07:01:09.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3132024-02-15T07:01:09.370ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
3142024-02-15T07:01:09.370ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 5.956518ms iteration = 7
3152024-02-15T07:01:09.371ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
3162024-02-15T07:01:09.371ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3172024-02-15T07:01:09.372ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
3182024-02-15T07:01:09.372ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3192024-02-15T07:01:09.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3202024-02-15T07:01:09.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3212024-02-15T07:01:09.372ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 7.550952ms iteration = 12
3222024-02-15T07:01:09.372ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
3232024-02-15T07:01:09.372ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3242024-02-15T07:01:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3252024-02-15T07:01:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
3262024-02-15T07:01:09.373ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3272024-02-15T07:01:09.373ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3282024-02-15T07:01:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3292024-02-15T07:01:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3302024-02-15T07:01:09.374ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3312024-02-15T07:01:09.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3322024-02-15T07:01:09.374ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 9.500764ms iteration = 32
3332024-02-15T07:01:09.374ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 12 reason = Timeout
3342024-02-15T07:01:09.374ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
3352024-02-15T07:01:09.374ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
3362024-02-15T07:01:09.374ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 9.19\u{b5}s iteration = 12
3372024-02-15T07:01:09.374ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3382024-02-15T07:01:09.374ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3392024-02-15T07:01:09.375ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
3402024-02-15T07:01:09.375ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = d7f8cbe9-558e-41ea-b105-dcc67f70cc11 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
3412024-02-15T07:01:09.375ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3422024-02-15T07:01:09.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3432024-02-15T07:01:09.375ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
3442024-02-15T07:01:09.375ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
3452024-02-15T07:01:09.375ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
3462024-02-15T07:01:09.375ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3472024-02-15T07:01:09.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3482024-02-15T07:01:09.376ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3492024-02-15T07:01:09.376ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3502024-02-15T07:01:09.378ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
3512024-02-15T07:01:09.378ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3522024-02-15T07:01:09.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3532024-02-15T07:01:09.380ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3542024-02-15T07:01:09.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3552024-02-15T07:01:09.381ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3562024-02-15T07:01:09.381ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3572024-02-15T07:01:09.381ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
3582024-02-15T07:01:09.381ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3592024-02-15T07:01:09.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3602024-02-15T07:01:09.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3612024-02-15T07:01:09.381ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3622024-02-15T07:01:09.383ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
3632024-02-15T07:01:09.383ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = c0170a00-b2b4-4879-8162-cfda0a5009a3 dns_zone_name = oxide.test found_so_far = 0 version = 2
3642024-02-15T07:01:09.383ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3652024-02-15T07:01:09.383ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = d7f8cbe9-558e-41ea-b105-dcc67f70cc11 dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
3662024-02-15T07:01:09.383ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
3672024-02-15T07:01:09.383ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-02-15 06:56:35.135865 UTC dns_group = internal generation = 1
3682024-02-15T07:01:09.383ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 25.764093ms iteration = 7
3692024-02-15T07:01:09.383ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3702024-02-15T07:01:09.383ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
3712024-02-15T07:01:09.383ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3722024-02-15T07:01:09.383ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3732024-02-15T07:01:09.384ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3742024-02-15T07:01:09.387ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3752024-02-15T07:01:09.387ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3762024-02-15T07:01:09.387ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3772024-02-15T07:01:09.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3782024-02-15T07:01:09.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3792024-02-15T07:01:09.387ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = c0170a00-b2b4-4879-8162-cfda0a5009a3 dns_zone_name = oxide.test found_so_far = 1 version = 2
3802024-02-15T07:01:09.387ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
3812024-02-15T07:01:09.387ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-02-15 06:56:35.417009 UTC dns_group = external generation = 2
3822024-02-15T07:01:09.387ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 23.116707ms iteration = 7
3832024-02-15T07:01:09.388ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3842024-02-15T07:01:09.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3852024-02-15T07:01:09.389ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3862024-02-15T07:01:09.392ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ec8d62a8-6b82-4888-b333-60568d39e395, lookup_type: ById(ec8d62a8-6b82-4888-b333-60568d39e395) } result = Ok(())
3872024-02-15T07:01:09.392ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3882024-02-15T07:01:09.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3892024-02-15T07:01:09.392ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-02-15T06:56:09.420632Z, time_modified: 2024-02-15T06:56:09.420632Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ffb05f40-4544-4250-8e84-31ffead32a4d, db_silo: Silo { identity: SiloIdentity { id: ffb05f40-4544-4250-8e84-31ffead32a4d, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T06:56:35.393136Z, time_modified: 2024-02-15T06:56:35.393136Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc23126eba6ce80ab4599432cb1e3814af0b1a2121110ea710e2bb25f380b4e9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ffb05f40-4544-4250-8e84-31ffead32a4d, db_silo: Silo { identity: SiloIdentity { id: ffb05f40-4544-4250-8e84-31ffead32a4d, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T06:56:35.393136Z, time_modified: 2024-02-15T06:56:35.393136Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc23126eba6ce80ab4599432cb1e3814af0b1a2121110ea710e2bb25f380b4e9" }] }) }
3902024-02-15T07:01:09.392ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 27.989431ms iteration = 7
3912024-02-15T07:01:09.393ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 28.776578ms iteration = 12
3922024-02-15T07:01:09.410ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
3932024-02-15T07:01:09.418ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
3942024-02-15T07:01:09.451ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
3952024-02-15T07:01:09.451ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "59ac94cd-b632-4143-bcee-f4a078f90d69", "date": "Thu, 15 Feb 2024 07:01:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
3962024-02-15T07:01:09.451ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
3972024-02-15T07:01:09.453ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "6fa6576b-a6c7-4dbd-8b8e-6401b042eb35", "date": "Thu, 15 Feb 2024 07:01:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
3982024-02-15T07:01:09.453ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
3992024-02-15T07:01:09.454ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
4002024-02-15T07:01:09.454ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 89.869379ms iteration = 9
4012024-02-15T07:01:09.461ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
4022024-02-15T07:01:09.491ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
4032024-02-15T07:01:09.491ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "755d206a-c68e-460e-9eb4-9a8d4e8a3459", "date": "Thu, 15 Feb 2024 07:01:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
4042024-02-15T07:01:09.491ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
4052024-02-15T07:01:09.493ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "4602e5f6-69ba-4a7d-8250-43d1ee24ea62", "date": "Thu, 15 Feb 2024 07:01:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
4062024-02-15T07:01:09.493ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
4072024-02-15T07:01:09.494ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "314d3f6d-ea87-4edc-ad7b-43e2c3f8c9ef", "date": "Thu, 15 Feb 2024 07:01:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
4082024-02-15T07:01:09.494ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
4092024-02-15T07:01:09.496ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
4102024-02-15T07:01:09.496ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 131.296269ms iteration = 9
4112024-02-15T07:01:09.518ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = d5a8d948-fb88-4bc9-a276-a9657557c623 roles = RoleSet { roles: {} } uri = /nat/ipv4/changeset/11?limit=100
4122024-02-15T07:01:09.518ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = d5a8d948-fb88-4bc9-a276-a9657557c623 resource = Database result = Ok(()) uri = /nat/ipv4/changeset/11?limit=100
4132024-02-15T07:01:09.521ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 2674 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = d5a8d948-fb88-4bc9-a276-a9657557c623 response_code = 200 uri = /nat/ipv4/changeset/11?limit=100
4142024-02-15T07:01:09.683ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 684 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 8059337d-43dd-4b1e-8823-772dbdda0226 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
4152024-02-15T07:01:19.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 33 reason = Timeout
4162024-02-15T07:01:19.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4172024-02-15T07:01:19.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4182024-02-15T07:01:19.368ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4192024-02-15T07:01:19.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4202024-02-15T07:01:19.370ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4212024-02-15T07:01:19.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4222024-02-15T07:01:19.372ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.541561ms iteration = 33
4232024-02-15T07:01:19.684ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 873 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 3316e625-8075-4f17-947a-8949a9742246 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
4242024-02-15T07:01:29.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 34 reason = Timeout
4252024-02-15T07:01:29.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4262024-02-15T07:01:29.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4272024-02-15T07:01:29.366ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4282024-02-15T07:01:29.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4292024-02-15T07:01:29.369ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4302024-02-15T07:01:29.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4312024-02-15T07:01:29.370ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.57865ms iteration = 34
4322024-02-15T07:01:29.683ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 590 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 48fd865b-b805-4afb-96e4-4e37361639c8 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
4332024-02-15T07:01:39.364ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 13 reason = Timeout
4342024-02-15T07:01:39.364ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
4352024-02-15T07:01:39.364ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 13 reason = Timeout
4362024-02-15T07:01:39.364ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
4372024-02-15T07:01:39.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
4382024-02-15T07:01:39.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
4392024-02-15T07:01:39.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 35 reason = Timeout
4402024-02-15T07:01:39.365ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 13 reason = Timeout
4412024-02-15T07:01:39.365ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
4422024-02-15T07:01:39.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4432024-02-15T07:01:39.365ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 13 reason = Timeout
4442024-02-15T07:01:39.365ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4452024-02-15T07:01:39.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4462024-02-15T07:01:39.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4472024-02-15T07:01:39.366ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
4482024-02-15T07:01:39.366ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
4492024-02-15T07:01:39.366ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9ec3b589-75e3-450e-8895-9500051e5e24", "content-length": "2", "date": "Thu, 15 Feb 2024 07:01:39 GMT"} })
4502024-02-15T07:01:39.366ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
4512024-02-15T07:01:39.366ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f99f0709-bbe4-4830-9af5-b85f74045fd7", "content-length": "2", "date": "Thu, 15 Feb 2024 07:01:39 GMT"} })
4522024-02-15T07:01:39.366ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 1.990732ms iteration = 13
4532024-02-15T07:01:39.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
4542024-02-15T07:01:39.367ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4552024-02-15T07:01:39.367ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4562024-02-15T07:01:39.368ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
4572024-02-15T07:01:39.368ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.453403ms iteration = 13
4582024-02-15T07:01:39.368ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 3.649499ms iteration = 13
4592024-02-15T07:01:39.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
4602024-02-15T07:01:39.370ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4612024-02-15T07:01:39.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4622024-02-15T07:01:39.370ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4632024-02-15T07:01:39.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4642024-02-15T07:01:39.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4652024-02-15T07:01:39.371ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.899417ms iteration = 35
4662024-02-15T07:01:39.371ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4672024-02-15T07:01:39.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4682024-02-15T07:01:39.374ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 13 reason = Timeout
4692024-02-15T07:01:39.374ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
4702024-02-15T07:01:39.374ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
4712024-02-15T07:01:39.374ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 10.82\u{b5}s iteration = 13
4722024-02-15T07:01:39.393ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4732024-02-15T07:01:39.394ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4742024-02-15T07:01:39.395ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4752024-02-15T07:01:39.395ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4762024-02-15T07:01:39.396ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4772024-02-15T07:01:39.399ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ec8d62a8-6b82-4888-b333-60568d39e395, lookup_type: ById(ec8d62a8-6b82-4888-b333-60568d39e395) } result = Ok(())
4782024-02-15T07:01:39.399ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4792024-02-15T07:01:39.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4802024-02-15T07:01:39.400ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 34.88756ms iteration = 13
4812024-02-15T07:01:39.550ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 4a0c3254-db33-4e58-9b9c-fbaedbe437a8 roles = RoleSet { roles: {} } uri = /nat/ipv4/changeset/11?limit=100
4822024-02-15T07:01:39.551ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 4a0c3254-db33-4e58-9b9c-fbaedbe437a8 resource = Database result = Ok(()) uri = /nat/ipv4/changeset/11?limit=100
4832024-02-15T07:01:39.553ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 2946 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 4a0c3254-db33-4e58-9b9c-fbaedbe437a8 response_code = 200 uri = /nat/ipv4/changeset/11?limit=100
4842024-02-15T07:01:39.684ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 844 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = ebd1c860-f44c-4e50-9549-6155c3b90261 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
4852024-02-15T07:01:49.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 36 reason = Timeout
4862024-02-15T07:01:49.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4872024-02-15T07:01:49.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4882024-02-15T07:01:49.367ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4892024-02-15T07:01:49.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4902024-02-15T07:01:49.369ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4912024-02-15T07:01:49.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4922024-02-15T07:01:49.371ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.874486ms iteration = 36
4932024-02-15T07:01:49.683ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 551 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 0aeb6d62-1891-4f67-9517-424038845cb7 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
4942024-02-15T07:01:59.364ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 37 reason = Timeout
4952024-02-15T07:01:59.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4962024-02-15T07:01:59.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4972024-02-15T07:01:59.366ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4982024-02-15T07:01:59.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4992024-02-15T07:01:59.369ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5002024-02-15T07:01:59.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5012024-02-15T07:01:59.370ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.528529ms iteration = 37
5022024-02-15T07:01:59.683ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 578 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 2008d2dc-a396-48f4-aeac-d399217f9841 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
5032024-02-15T07:02:09.357ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
5042024-02-15T07:02:09.357ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
5052024-02-15T07:02:09.357ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5062024-02-15T07:02:09.357ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
5072024-02-15T07:02:09.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5082024-02-15T07:02:09.358ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5092024-02-15T07:02:09.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5102024-02-15T07:02:09.359ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("78a9790e-f724-40b6-bd7f-e718315a779a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("de8bbd15-7094-44ed-a766-1edae8657a5e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ffdb0d80-1b9f-4db4-a0b6-96d3f7810c28.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ffdb0d80-1b9f-4db4-a0b6-96d3f7810c28.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 1956, tv_nsec: 185338104 } })
5112024-02-15T07:02:09.359ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5122024-02-15T07:02:09.363ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
5132024-02-15T07:02:09.363ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
5142024-02-15T07:02:09.363ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
5152024-02-15T07:02:09.363ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5162024-02-15T07:02:09.363ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
5172024-02-15T07:02:09.363ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
5182024-02-15T07:02:09.363ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
5192024-02-15T07:02:09.363ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
5202024-02-15T07:02:09.363ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
5212024-02-15T07:02:09.363ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
5222024-02-15T07:02:09.363ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
5232024-02-15T07:02:09.363ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5242024-02-15T07:02:09.364ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
5252024-02-15T07:02:09.364ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 6.99469ms iteration = 8
5262024-02-15T07:02:09.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
5272024-02-15T07:02:09.364ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
5282024-02-15T07:02:09.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
5292024-02-15T07:02:09.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5302024-02-15T07:02:09.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5312024-02-15T07:02:09.364ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5322024-02-15T07:02:09.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5332024-02-15T07:02:09.365ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
5342024-02-15T07:02:09.365ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
5352024-02-15T07:02:09.365ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
5362024-02-15T07:02:09.365ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5372024-02-15T07:02:09.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 38 reason = Timeout
5382024-02-15T07:02:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5392024-02-15T07:02:09.365ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
5402024-02-15T07:02:09.365ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
5412024-02-15T07:02:09.365ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
5422024-02-15T07:02:09.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5432024-02-15T07:02:09.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 8 reason = Timeout
5442024-02-15T07:02:09.365ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5452024-02-15T07:02:09.365ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:46 reason = no blueprint
5462024-02-15T07:02:09.365ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 5.4\u{b5}s iteration = 8
5472024-02-15T07:02:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5482024-02-15T07:02:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5492024-02-15T07:02:09.365ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
5502024-02-15T07:02:09.366ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
5512024-02-15T07:02:09.366ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d363611b-ac31-4645-b946-87293210fc2e", "content-length": "2", "date": "Thu, 15 Feb 2024 07:02:09 GMT"} })
5522024-02-15T07:02:09.366ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
5532024-02-15T07:02:09.366ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5542024-02-15T07:02:09.366ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("19feddd6-78c6-4e31-b93a-acace968955e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("32bd8fe1-df5d-4887-b6be-0cddecf4285c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("32bd8fe1-df5d-4887-b6be-0cddecf4285c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::9)) }], valid_until: Instant { tv_sec: 1956, tv_nsec: 192892836 } })
5552024-02-15T07:02:09.366ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5562024-02-15T07:02:09.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
5572024-02-15T07:02:09.367ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5582024-02-15T07:02:09.367ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5592024-02-15T07:02:09.367ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5602024-02-15T07:02:09.367ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "66c19c15-dd0a-4fdb-9cc4-0ff8c9c1197a", "content-length": "2", "date": "Thu, 15 Feb 2024 07:02:09 GMT"} })
5612024-02-15T07:02:09.367ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 3.967396ms iteration = 14
5622024-02-15T07:02:09.367ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
5632024-02-15T07:02:09.367ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 4.078218ms iteration = 8
5642024-02-15T07:02:09.368ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 4.790743ms iteration = 14
5652024-02-15T07:02:09.369ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
5662024-02-15T07:02:09.369ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.004415ms iteration = 14
5672024-02-15T07:02:09.371ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5682024-02-15T07:02:09.372ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
5692024-02-15T07:02:09.372ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
5702024-02-15T07:02:09.372ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
5712024-02-15T07:02:09.372ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
5722024-02-15T07:02:09.372ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5732024-02-15T07:02:09.372ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5742024-02-15T07:02:09.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5752024-02-15T07:02:09.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5762024-02-15T07:02:09.372ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
5772024-02-15T07:02:09.372ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5782024-02-15T07:02:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5792024-02-15T07:02:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5802024-02-15T07:02:09.373ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5812024-02-15T07:02:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5822024-02-15T07:02:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
5832024-02-15T07:02:09.373ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5842024-02-15T07:02:09.374ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5852024-02-15T07:02:09.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5862024-02-15T07:02:09.374ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
5872024-02-15T07:02:09.374ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = d7f8cbe9-558e-41ea-b105-dcc67f70cc11 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
5882024-02-15T07:02:09.374ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
5892024-02-15T07:02:09.374ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
5902024-02-15T07:02:09.374ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
5912024-02-15T07:02:09.374ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5922024-02-15T07:02:09.374ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 5.4\u{b5}s iteration = 14
5932024-02-15T07:02:09.374ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5942024-02-15T07:02:09.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5952024-02-15T07:02:09.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5962024-02-15T07:02:09.374ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 9.624717ms iteration = 38
5972024-02-15T07:02:09.375ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5982024-02-15T07:02:09.375ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5992024-02-15T07:02:09.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6002024-02-15T07:02:09.376ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6012024-02-15T07:02:09.378ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
6022024-02-15T07:02:09.378ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6032024-02-15T07:02:09.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6042024-02-15T07:02:09.379ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
6052024-02-15T07:02:09.379ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6062024-02-15T07:02:09.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6072024-02-15T07:02:09.380ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
6082024-02-15T07:02:09.380ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = c0170a00-b2b4-4879-8162-cfda0a5009a3 dns_zone_name = oxide.test found_so_far = 0 version = 2
6092024-02-15T07:02:09.380ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6102024-02-15T07:02:09.380ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6112024-02-15T07:02:09.380ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6122024-02-15T07:02:09.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6132024-02-15T07:02:09.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6142024-02-15T07:02:09.381ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6152024-02-15T07:02:09.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6162024-02-15T07:02:09.382ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6172024-02-15T07:02:09.382ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = d7f8cbe9-558e-41ea-b105-dcc67f70cc11 dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
6182024-02-15T07:02:09.382ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6192024-02-15T07:02:09.382ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
6202024-02-15T07:02:09.382ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-02-15 06:56:35.135865 UTC dns_group = internal generation = 1
6212024-02-15T07:02:09.382ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 25.875146ms iteration = 8
6222024-02-15T07:02:09.385ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
6232024-02-15T07:02:09.385ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6242024-02-15T07:02:09.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6252024-02-15T07:02:09.386ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
6262024-02-15T07:02:09.386ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6272024-02-15T07:02:09.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6282024-02-15T07:02:09.387ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = c0170a00-b2b4-4879-8162-cfda0a5009a3 dns_zone_name = oxide.test found_so_far = 1 version = 2
6292024-02-15T07:02:09.387ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
6302024-02-15T07:02:09.387ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-02-15 06:56:35.417009 UTC dns_group = external generation = 2
6312024-02-15T07:02:09.387ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 23.181388ms iteration = 8
6322024-02-15T07:02:09.387ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6332024-02-15T07:02:09.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6342024-02-15T07:02:09.388ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6352024-02-15T07:02:09.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6362024-02-15T07:02:09.389ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6372024-02-15T07:02:09.392ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ec8d62a8-6b82-4888-b333-60568d39e395, lookup_type: ById(ec8d62a8-6b82-4888-b333-60568d39e395) } result = Ok(())
6382024-02-15T07:02:09.392ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6392024-02-15T07:02:09.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6402024-02-15T07:02:09.394ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 28.777128ms iteration = 14
6412024-02-15T07:02:09.395ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-02-15T06:56:09.420632Z, time_modified: 2024-02-15T06:56:09.420632Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ffb05f40-4544-4250-8e84-31ffead32a4d, db_silo: Silo { identity: SiloIdentity { id: ffb05f40-4544-4250-8e84-31ffead32a4d, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T06:56:35.393136Z, time_modified: 2024-02-15T06:56:35.393136Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc23126eba6ce80ab4599432cb1e3814af0b1a2121110ea710e2bb25f380b4e9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ffb05f40-4544-4250-8e84-31ffead32a4d, db_silo: Silo { identity: SiloIdentity { id: ffb05f40-4544-4250-8e84-31ffead32a4d, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T06:56:35.393136Z, time_modified: 2024-02-15T06:56:35.393136Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc23126eba6ce80ab4599432cb1e3814af0b1a2121110ea710e2bb25f380b4e9" }] }) }
6422024-02-15T07:02:09.395ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 31.491936ms iteration = 8
6432024-02-15T07:02:09.431ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
6442024-02-15T07:02:09.432ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
6452024-02-15T07:02:09.492ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
6462024-02-15T07:02:09.493ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "4534bcf8-7989-4472-ae1d-19079ecf255f", "date": "Thu, 15 Feb 2024 07:02:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
6472024-02-15T07:02:09.493ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
6482024-02-15T07:02:09.494ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
6492024-02-15T07:02:09.494ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "661033fd-978e-459f-98ce-81f3ae628e79", "date": "Thu, 15 Feb 2024 07:02:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
6502024-02-15T07:02:09.494ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
6512024-02-15T07:02:09.495ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
6522024-02-15T07:02:09.495ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 131.992745ms iteration = 10
6532024-02-15T07:02:09.524ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
6542024-02-15T07:02:09.524ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "d914608d-6105-48ff-bf71-a0cd5ef62940", "date": "Thu, 15 Feb 2024 07:02:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
6552024-02-15T07:02:09.524ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
6562024-02-15T07:02:09.526ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "064b4599-60e3-4083-823e-31f84aa769c0", "date": "Thu, 15 Feb 2024 07:02:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
6572024-02-15T07:02:09.526ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
6582024-02-15T07:02:09.528ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "19cc6e5d-3091-450d-9175-3afb698206fd", "date": "Thu, 15 Feb 2024 07:02:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
6592024-02-15T07:02:09.528ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
6602024-02-15T07:02:09.530ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
6612024-02-15T07:02:09.530ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 166.631599ms iteration = 10
6622024-02-15T07:02:09.584ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 11245987-7440-48b8-968c-0dc4f3bfca71 roles = RoleSet { roles: {} } uri = /nat/ipv4/changeset/11?limit=100
6632024-02-15T07:02:09.584ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 11245987-7440-48b8-968c-0dc4f3bfca71 resource = Database result = Ok(()) uri = /nat/ipv4/changeset/11?limit=100
6642024-02-15T07:02:09.587ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 3000 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 11245987-7440-48b8-968c-0dc4f3bfca71 response_code = 200 uri = /nat/ipv4/changeset/11?limit=100
6652024-02-15T07:02:09.684ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 675 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 0c6ccdd6-1351-43a3-bb6f-f65799e1be4f response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
6662024-02-15T07:02:19.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 39 reason = Timeout
6672024-02-15T07:02:19.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6682024-02-15T07:02:19.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6692024-02-15T07:02:19.366ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6702024-02-15T07:02:19.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
6712024-02-15T07:02:19.369ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6722024-02-15T07:02:19.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6732024-02-15T07:02:19.371ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.025399ms iteration = 39
6742024-02-15T07:02:19.683ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 722 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = d530ab17-187b-4170-82bc-32cd2cf05463 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
6752024-02-15T07:02:29.364ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
6762024-02-15T07:02:29.364ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6772024-02-15T07:02:29.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6782024-02-15T07:02:29.366ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6792024-02-15T07:02:29.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
6802024-02-15T07:02:29.369ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6812024-02-15T07:02:29.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6822024-02-15T07:02:29.371ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.262045ms iteration = 40
6832024-02-15T07:02:29.683ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 602 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 5c3e3565-1297-4ea4-8747-9da6ae2bc08f response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
6842024-02-15T07:02:39.363ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
6852024-02-15T07:02:39.364ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
6862024-02-15T07:02:39.364ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6872024-02-15T07:02:39.364ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
6882024-02-15T07:02:39.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6892024-02-15T07:02:39.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
6902024-02-15T07:02:39.364ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
6912024-02-15T07:02:39.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
6922024-02-15T07:02:39.365ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
6932024-02-15T07:02:39.365ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
6942024-02-15T07:02:39.365ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6952024-02-15T07:02:39.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6962024-02-15T07:02:39.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6972024-02-15T07:02:39.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6982024-02-15T07:02:39.365ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
6992024-02-15T07:02:39.365ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "fb11cc42-e9d0-4ae5-812f-eb26f97d4ac7", "content-length": "2", "date": "Thu, 15 Feb 2024 07:02:39 GMT"} })
7002024-02-15T07:02:39.365ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7012024-02-15T07:02:39.366ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
7022024-02-15T07:02:39.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7032024-02-15T07:02:39.366ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "06d3cb6e-3c22-4e57-a8ca-03e905882c32", "content-length": "2", "date": "Thu, 15 Feb 2024 07:02:39 GMT"} })
7042024-02-15T07:02:39.366ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 2.393131ms iteration = 15
7052024-02-15T07:02:39.366ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7062024-02-15T07:02:39.366ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7072024-02-15T07:02:39.367ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
7082024-02-15T07:02:39.367ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.414752ms iteration = 15
7092024-02-15T07:02:39.367ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 3.643138ms iteration = 15
7102024-02-15T07:02:39.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7112024-02-15T07:02:39.369ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7122024-02-15T07:02:39.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
7132024-02-15T07:02:39.369ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7142024-02-15T07:02:39.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7152024-02-15T07:02:39.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7162024-02-15T07:02:39.370ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.766144ms iteration = 41
7172024-02-15T07:02:39.371ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7182024-02-15T07:02:39.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7192024-02-15T07:02:39.374ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
7202024-02-15T07:02:39.374ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
7212024-02-15T07:02:39.374ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
7222024-02-15T07:02:39.374ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 10.78\u{b5}s iteration = 15
7232024-02-15T07:02:39.384ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7242024-02-15T07:02:39.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7252024-02-15T07:02:39.386ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7262024-02-15T07:02:39.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7272024-02-15T07:02:39.387ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7282024-02-15T07:02:39.390ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ec8d62a8-6b82-4888-b333-60568d39e395, lookup_type: ById(ec8d62a8-6b82-4888-b333-60568d39e395) } result = Ok(())
7292024-02-15T07:02:39.390ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7302024-02-15T07:02:39.390ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7312024-02-15T07:02:39.391ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 26.628162ms iteration = 15
7322024-02-15T07:02:39.620ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = e09a0261-ce07-4e10-ae48-19e922ee93c3 roles = RoleSet { roles: {} } uri = /nat/ipv4/changeset/11?limit=100
7332024-02-15T07:02:39.620ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = e09a0261-ce07-4e10-ae48-19e922ee93c3 resource = Database result = Ok(()) uri = /nat/ipv4/changeset/11?limit=100
7342024-02-15T07:02:39.623ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 3361 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = e09a0261-ce07-4e10-ae48-19e922ee93c3 response_code = 200 uri = /nat/ipv4/changeset/11?limit=100
7352024-02-15T07:02:39.624ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 7f303aaa-f0b2-40eb-87e8-8901799747e0 roles = RoleSet { roles: {} } uri = /nat/ipv4/changeset/13?limit=100
7362024-02-15T07:02:39.625ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 7f303aaa-f0b2-40eb-87e8-8901799747e0 resource = Database result = Ok(()) uri = /nat/ipv4/changeset/13?limit=100
7372024-02-15T07:02:39.627ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 2596 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 7f303aaa-f0b2-40eb-87e8-8901799747e0 response_code = 200 uri = /nat/ipv4/changeset/13?limit=100
7382024-02-15T07:02:39.683ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 520 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 755db690-cfc1-4050-b5ab-6c9d6f52cc00 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
7392024-02-15T07:02:49.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
7402024-02-15T07:02:49.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7412024-02-15T07:02:49.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7422024-02-15T07:02:49.366ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7432024-02-15T07:02:49.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7442024-02-15T07:02:49.369ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7452024-02-15T07:02:49.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7462024-02-15T07:02:49.370ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.773794ms iteration = 42
7472024-02-15T07:02:49.683ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 612 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 01109082-b7a0-4a63-970f-c819c2c5f902 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
7482024-02-15T07:02:59.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
7492024-02-15T07:02:59.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7502024-02-15T07:02:59.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7512024-02-15T07:02:59.367ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7522024-02-15T07:02:59.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7532024-02-15T07:02:59.370ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7542024-02-15T07:02:59.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7552024-02-15T07:02:59.371ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.234483ms iteration = 43
7562024-02-15T07:02:59.683ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 898 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 1ad6c47c-0e8c-4e0f-902f-1c0b7e174b5d response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
7572024-02-15T07:03:09.357ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
7582024-02-15T07:03:09.357ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
7592024-02-15T07:03:09.357ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7602024-02-15T07:03:09.357ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
7612024-02-15T07:03:09.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7622024-02-15T07:03:09.358ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7632024-02-15T07:03:09.358ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("78a9790e-f724-40b6-bd7f-e718315a779a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("de8bbd15-7094-44ed-a766-1edae8657a5e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ffdb0d80-1b9f-4db4-a0b6-96d3f7810c28.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ffdb0d80-1b9f-4db4-a0b6-96d3f7810c28.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 2016, tv_nsec: 184564276 } })
7642024-02-15T07:03:09.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7652024-02-15T07:03:09.360ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7662024-02-15T07:03:09.363ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
7672024-02-15T07:03:09.363ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 5.894577ms iteration = 9
7682024-02-15T07:03:09.363ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
7692024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
7702024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
7712024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7722024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
7732024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
7742024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
7752024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
7762024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
7772024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
7782024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
7792024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
7802024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
7812024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
7822024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7832024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
7842024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
7852024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
7862024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
7872024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
7882024-02-15T07:03:09.364ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7892024-02-15T07:03:09.365ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
7902024-02-15T07:03:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7912024-02-15T07:03:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
7922024-02-15T07:03:09.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
7932024-02-15T07:03:09.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7942024-02-15T07:03:09.365ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
7952024-02-15T07:03:09.365ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
7962024-02-15T07:03:09.365ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
7972024-02-15T07:03:09.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
7982024-02-15T07:03:09.365ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:46 reason = no blueprint
7992024-02-15T07:03:09.365ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8002024-02-15T07:03:09.365ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 5.79\u{b5}s iteration = 9
8012024-02-15T07:03:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8022024-02-15T07:03:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8032024-02-15T07:03:09.365ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8042024-02-15T07:03:09.366ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
8052024-02-15T07:03:09.366ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
8062024-02-15T07:03:09.366ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9d5ea5ff-ac39-4826-a17b-166ad906dfdb", "content-length": "2", "date": "Thu, 15 Feb 2024 07:03:09 GMT"} })
8072024-02-15T07:03:09.366ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 2.82121ms iteration = 16
8082024-02-15T07:03:09.367ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "441e342d-4c4a-42fc-883d-7455e666b781", "content-length": "2", "date": "Thu, 15 Feb 2024 07:03:09 GMT"} })
8092024-02-15T07:03:09.367ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
8102024-02-15T07:03:09.367ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8112024-02-15T07:03:09.367ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("19feddd6-78c6-4e31-b93a-acace968955e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("32bd8fe1-df5d-4887-b6be-0cddecf4285c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("32bd8fe1-df5d-4887-b6be-0cddecf4285c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::9)) }], valid_until: Instant { tv_sec: 2016, tv_nsec: 193673591 } })
8122024-02-15T07:03:09.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
8132024-02-15T07:03:09.368ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
8142024-02-15T07:03:09.368ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8152024-02-15T07:03:09.368ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
8162024-02-15T07:03:09.368ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 3.979486ms iteration = 9
8172024-02-15T07:03:09.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8182024-02-15T07:03:09.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8192024-02-15T07:03:09.370ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8202024-02-15T07:03:09.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8212024-02-15T07:03:09.373ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8222024-02-15T07:03:09.373ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
8232024-02-15T07:03:09.373ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
8242024-02-15T07:03:09.373ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
8252024-02-15T07:03:09.373ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 9.99\u{b5}s iteration = 16
8262024-02-15T07:03:09.373ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8272024-02-15T07:03:09.373ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8282024-02-15T07:03:09.375ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 10.586247ms iteration = 44
8292024-02-15T07:03:09.376ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8302024-02-15T07:03:09.376ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 12.185962ms iteration = 16
8312024-02-15T07:03:09.376ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
8322024-02-15T07:03:09.376ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 10.936635ms iteration = 16
8332024-02-15T07:03:09.376ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8342024-02-15T07:03:09.377ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
8352024-02-15T07:03:09.377ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8362024-02-15T07:03:09.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8372024-02-15T07:03:09.377ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
8382024-02-15T07:03:09.377ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8392024-02-15T07:03:09.378ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
8402024-02-15T07:03:09.378ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8412024-02-15T07:03:09.378ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
8422024-02-15T07:03:09.378ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
8432024-02-15T07:03:09.378ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
8442024-02-15T07:03:09.378ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8452024-02-15T07:03:09.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8462024-02-15T07:03:09.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8472024-02-15T07:03:09.378ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8482024-02-15T07:03:09.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8492024-02-15T07:03:09.379ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8502024-02-15T07:03:09.379ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
8512024-02-15T07:03:09.379ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = d7f8cbe9-558e-41ea-b105-dcc67f70cc11 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
8522024-02-15T07:03:09.379ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8532024-02-15T07:03:09.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8542024-02-15T07:03:09.381ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8552024-02-15T07:03:09.381ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
8562024-02-15T07:03:09.381ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8572024-02-15T07:03:09.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8582024-02-15T07:03:09.383ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8592024-02-15T07:03:09.383ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8602024-02-15T07:03:09.384ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
8612024-02-15T07:03:09.384ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8622024-02-15T07:03:09.384ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8632024-02-15T07:03:09.384ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
8642024-02-15T07:03:09.384ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8652024-02-15T07:03:09.384ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8662024-02-15T07:03:09.384ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8672024-02-15T07:03:09.386ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = d7f8cbe9-558e-41ea-b105-dcc67f70cc11 dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
8682024-02-15T07:03:09.386ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
8692024-02-15T07:03:09.386ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-02-15 06:56:35.135865 UTC dns_group = internal generation = 1
8702024-02-15T07:03:09.386ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 29.078365ms iteration = 9
8712024-02-15T07:03:09.386ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
8722024-02-15T07:03:09.386ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = c0170a00-b2b4-4879-8162-cfda0a5009a3 dns_zone_name = oxide.test found_so_far = 0 version = 2
8732024-02-15T07:03:09.386ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8742024-02-15T07:03:09.386ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
8752024-02-15T07:03:09.386ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8762024-02-15T07:03:09.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8772024-02-15T07:03:09.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8782024-02-15T07:03:09.388ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8792024-02-15T07:03:09.390ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8802024-02-15T07:03:09.390ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
8812024-02-15T07:03:09.390ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8822024-02-15T07:03:09.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8832024-02-15T07:03:09.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8842024-02-15T07:03:09.392ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = c0170a00-b2b4-4879-8162-cfda0a5009a3 dns_zone_name = oxide.test found_so_far = 1 version = 2
8852024-02-15T07:03:09.392ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
8862024-02-15T07:03:09.392ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-02-15 06:56:35.417009 UTC dns_group = external generation = 2
8872024-02-15T07:03:09.392ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8882024-02-15T07:03:09.392ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 27.992842ms iteration = 9
8892024-02-15T07:03:09.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8902024-02-15T07:03:09.392ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-02-15T06:56:09.420632Z, time_modified: 2024-02-15T06:56:09.420632Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ffb05f40-4544-4250-8e84-31ffead32a4d, db_silo: Silo { identity: SiloIdentity { id: ffb05f40-4544-4250-8e84-31ffead32a4d, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T06:56:35.393136Z, time_modified: 2024-02-15T06:56:35.393136Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "bc23126eba6ce80ab4599432cb1e3814af0b1a2121110ea710e2bb25f380b4e9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ffb05f40-4544-4250-8e84-31ffead32a4d, db_silo: Silo { identity: SiloIdentity { id: ffb05f40-4544-4250-8e84-31ffead32a4d, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T06:56:35.393136Z, time_modified: 2024-02-15T06:56:35.393136Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "bc23126eba6ce80ab4599432cb1e3814af0b1a2121110ea710e2bb25f380b4e9" }] }) } file = nexus/src/app/background/external_endpoints.rs:101
8912024-02-15T07:03:09.392ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 28.367159ms iteration = 9
8922024-02-15T07:03:09.393ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8932024-02-15T07:03:09.395ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ec8d62a8-6b82-4888-b333-60568d39e395, lookup_type: ById(ec8d62a8-6b82-4888-b333-60568d39e395) } result = Ok(())
8942024-02-15T07:03:09.395ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8952024-02-15T07:03:09.396ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8962024-02-15T07:03:09.397ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 31.680391ms iteration = 16
8972024-02-15T07:03:09.490ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
8982024-02-15T07:03:09.490ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
8992024-02-15T07:03:09.606ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
9002024-02-15T07:03:09.607ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
9012024-02-15T07:03:09.607ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "17bdc29a-ad6f-4ac1-9c54-6697550c452c", "date": "Thu, 15 Feb 2024 07:03:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
9022024-02-15T07:03:09.607ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
9032024-02-15T07:03:09.609ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "6f57e4ec-ad71-4dfb-a099-79eeea3f404b", "date": "Thu, 15 Feb 2024 07:03:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
9042024-02-15T07:03:09.609ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
9052024-02-15T07:03:09.610ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
9062024-02-15T07:03:09.610ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 246.31303ms iteration = 11
9072024-02-15T07:03:09.637ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
9082024-02-15T07:03:09.637ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "ab65aa62-1b91-434d-862d-ac7341a07ff1", "date": "Thu, 15 Feb 2024 07:03:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
9092024-02-15T07:03:09.637ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
9102024-02-15T07:03:09.639ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "44631dd1-7913-4ae5-a43e-1d71478f092c", "date": "Thu, 15 Feb 2024 07:03:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
9112024-02-15T07:03:09.639ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
9122024-02-15T07:03:09.641ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "04601301-fb00-4da7-a9ba-17842e443394", "date": "Thu, 15 Feb 2024 07:03:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
9132024-02-15T07:03:09.641ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
9142024-02-15T07:03:09.643ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
9152024-02-15T07:03:09.643ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 279.055453ms iteration = 11
9162024-02-15T07:03:09.659ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 80853e22-5d92-41bc-9c64-ea7762abe0f7 roles = RoleSet { roles: {} } uri = /nat/ipv4/changeset/13?limit=100
9172024-02-15T07:03:09.659ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 80853e22-5d92-41bc-9c64-ea7762abe0f7 resource = Database result = Ok(()) uri = /nat/ipv4/changeset/13?limit=100
9182024-02-15T07:03:09.663ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 3946 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 80853e22-5d92-41bc-9c64-ea7762abe0f7 response_code = 200 uri = /nat/ipv4/changeset/13?limit=100
9192024-02-15T07:03:09.664ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = be5e7e20-f163-4737-822b-868bd3c7d099 roles = RoleSet { roles: {} } uri = /nat/ipv4/changeset/15?limit=100
9202024-02-15T07:03:09.665ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = be5e7e20-f163-4737-822b-868bd3c7d099 resource = Database result = Ok(()) uri = /nat/ipv4/changeset/15?limit=100
9212024-02-15T07:03:09.667ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 3082 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = be5e7e20-f163-4737-822b-868bd3c7d099 response_code = 200 uri = /nat/ipv4/changeset/15?limit=100
9222024-02-15T07:03:09.683ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 649 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = dc1d34c6-538c-43ec-8636-a0d372937e94 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
9232024-02-15T07:03:19.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
9242024-02-15T07:03:19.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9252024-02-15T07:03:19.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9262024-02-15T07:03:19.367ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9272024-02-15T07:03:19.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9282024-02-15T07:03:19.369ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9292024-02-15T07:03:19.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9302024-02-15T07:03:19.371ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.504878ms iteration = 45
9312024-02-15T07:03:19.684ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 1214 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 9d4323ba-5218-4749-8026-3779b90650ab response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
9322024-02-15T07:03:29.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
9332024-02-15T07:03:29.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9342024-02-15T07:03:29.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9352024-02-15T07:03:29.366ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9362024-02-15T07:03:29.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9372024-02-15T07:03:29.369ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9382024-02-15T07:03:29.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9392024-02-15T07:03:29.370ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.806025ms iteration = 46
9402024-02-15T07:03:29.683ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 522 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 68daef3e-01d3-44d2-8683-8278c07bb3b9 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
9412024-02-15T07:03:39.364ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
9422024-02-15T07:03:39.364ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
9432024-02-15T07:03:39.364ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
9442024-02-15T07:03:39.364ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
9452024-02-15T07:03:39.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
9462024-02-15T07:03:39.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
9472024-02-15T07:03:39.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
9482024-02-15T07:03:39.365ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
9492024-02-15T07:03:39.365ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
9502024-02-15T07:03:39.365ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
9512024-02-15T07:03:39.365ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9522024-02-15T07:03:39.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9532024-02-15T07:03:39.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9542024-02-15T07:03:39.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9552024-02-15T07:03:39.366ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
9562024-02-15T07:03:39.366ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "571efe13-a0ae-4608-a6a0-a22a2391bec1", "content-length": "2", "date": "Thu, 15 Feb 2024 07:03:39 GMT"} })
9572024-02-15T07:03:39.366ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
9582024-02-15T07:03:39.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
9592024-02-15T07:03:39.366ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
9602024-02-15T07:03:39.366ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "80b65daa-3751-4f24-8947-2077b02f9c65", "content-length": "2", "date": "Thu, 15 Feb 2024 07:03:39 GMT"} })
9612024-02-15T07:03:39.366ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 2.31361ms iteration = 17
9622024-02-15T07:03:39.367ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9632024-02-15T07:03:39.367ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9642024-02-15T07:03:39.368ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 3.71441ms iteration = 17
9652024-02-15T07:03:39.368ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
9662024-02-15T07:03:39.368ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.793ms iteration = 17
9672024-02-15T07:03:39.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9682024-02-15T07:03:39.370ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9692024-02-15T07:03:39.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9702024-02-15T07:03:39.370ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9712024-02-15T07:03:39.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9722024-02-15T07:03:39.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9732024-02-15T07:03:39.372ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.623822ms iteration = 47
9742024-02-15T07:03:39.372ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9752024-02-15T07:03:39.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9762024-02-15T07:03:39.374ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
9772024-02-15T07:03:39.374ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
9782024-02-15T07:03:39.374ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
9792024-02-15T07:03:39.374ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 9.291\u{b5}s iteration = 17
9802024-02-15T07:03:39.383ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9812024-02-15T07:03:39.383ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9822024-02-15T07:03:39.384ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9832024-02-15T07:03:39.384ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9842024-02-15T07:03:39.385ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9852024-02-15T07:03:39.388ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ec8d62a8-6b82-4888-b333-60568d39e395, lookup_type: ById(ec8d62a8-6b82-4888-b333-60568d39e395) } result = Ok(())
9862024-02-15T07:03:39.388ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9872024-02-15T07:03:39.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9882024-02-15T07:03:39.390ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 24.374304ms iteration = 17
9892024-02-15T07:03:39.683ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 933 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 6b8fc076-8adb-4080-98b4-b3e1df9ccb03 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
9902024-02-15T07:03:39.696ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = e2c15374-689f-45a3-aa05-520e93d6c4ba roles = RoleSet { roles: {} } uri = /nat/ipv4/changeset/15?limit=100
9912024-02-15T07:03:39.697ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = e2c15374-689f-45a3-aa05-520e93d6c4ba resource = Database result = Ok(()) uri = /nat/ipv4/changeset/15?limit=100
9922024-02-15T07:03:39.700ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 3521 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = e2c15374-689f-45a3-aa05-520e93d6c4ba response_code = 200 uri = /nat/ipv4/changeset/15?limit=100
9932024-02-15T07:03:49.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
9942024-02-15T07:03:49.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9952024-02-15T07:03:49.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9962024-02-15T07:03:49.366ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9972024-02-15T07:03:49.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9982024-02-15T07:03:49.369ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9992024-02-15T07:03:49.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10002024-02-15T07:03:49.370ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.664102ms iteration = 48
10012024-02-15T07:03:49.684ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 999 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = 48f2feb2-a76e-43ba-a74a-fbb4b4d91b61 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
10022024-02-15T07:03:59.364ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
10032024-02-15T07:03:59.364ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10042024-02-15T07:03:59.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10052024-02-15T07:03:59.366ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10062024-02-15T07:03:59.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10072024-02-15T07:03:59.369ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10082024-02-15T07:03:59.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10092024-02-15T07:03:59.370ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.017559ms iteration = 49
10102024-02-15T07:03:59.684ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 712 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = dccccbf3-844e-4525-beea-abc1e1c12841 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
10112024-02-15T07:04:09.357ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 10 reason = Timeout
10122024-02-15T07:04:09.357ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
10132024-02-15T07:04:09.357ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
10142024-02-15T07:04:09.357ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 10 reason = Timeout
10152024-02-15T07:04:09.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
10162024-02-15T07:04:09.358ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
10172024-02-15T07:04:09.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
10182024-02-15T07:04:09.360ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10192024-02-15T07:04:09.361ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("78a9790e-f724-40b6-bd7f-e718315a779a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("de8bbd15-7094-44ed-a766-1edae8657a5e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ffdb0d80-1b9f-4db4-a0b6-96d3f7810c28.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ffdb0d80-1b9f-4db4-a0b6-96d3f7810c28.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 2076, tv_nsec: 187611170 } })
10202024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
10212024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
10222024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 10 reason = Timeout
10232024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 10 reason = Timeout
10242024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10252024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 10 reason = Timeout
10262024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 12 reason = Timeout
10272024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
10282024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10292024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
10302024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
10312024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 12 reason = Timeout
10322024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
10332024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
10342024-02-15T07:04:09.364ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
10352024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10362024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
10372024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
10382024-02-15T07:04:09.364ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:46 reason = no blueprint
10392024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.87\u{b5}s iteration = 10
10402024-02-15T07:04:09.364ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10412024-02-15T07:04:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
10422024-02-15T07:04:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10432024-02-15T07:04:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10442024-02-15T07:04:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10452024-02-15T07:04:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
10462024-02-15T07:04:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10472024-02-15T07:04:09.365ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
10482024-02-15T07:04:09.366ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10492024-02-15T07:04:09.366ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
10502024-02-15T07:04:09.366ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
10512024-02-15T07:04:09.366ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
10522024-02-15T07:04:09.366ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
10532024-02-15T07:04:09.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10542024-02-15T07:04:09.366ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
10552024-02-15T07:04:09.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
10562024-02-15T07:04:09.367ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
10572024-02-15T07:04:09.367ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "128a929f-4389-45bc-bf94-968f7ad95362", "content-length": "2", "date": "Thu, 15 Feb 2024 07:04:09 GMT"} })
10582024-02-15T07:04:09.367ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10592024-02-15T07:04:09.367ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
10602024-02-15T07:04:09.367ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "41a5649d-c4f4-4f99-aa1b-a872e580ed71", "content-length": "2", "date": "Thu, 15 Feb 2024 07:04:09 GMT"} })
10612024-02-15T07:04:09.367ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 2.905293ms iteration = 18
10622024-02-15T07:04:09.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
10632024-02-15T07:04:09.368ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10642024-02-15T07:04:09.368ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10652024-02-15T07:04:09.368ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10662024-02-15T07:04:09.368ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10672024-02-15T07:04:09.368ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
10682024-02-15T07:04:09.368ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
10692024-02-15T07:04:09.368ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 11.1626ms iteration = 10
10702024-02-15T07:04:09.368ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.111108ms iteration = 18
10712024-02-15T07:04:09.368ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("19feddd6-78c6-4e31-b93a-acace968955e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("32bd8fe1-df5d-4887-b6be-0cddecf4285c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("32bd8fe1-df5d-4887-b6be-0cddecf4285c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::9)) }], valid_until: Instant { tv_sec: 2076, tv_nsec: 195316946 } })
10722024-02-15T07:04:09.369ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 5.279784ms iteration = 18
10732024-02-15T07:04:09.370ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
10742024-02-15T07:04:09.370ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 5.672902ms iteration = 10
10752024-02-15T07:04:09.372ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
10762024-02-15T07:04:09.372ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10772024-02-15T07:04:09.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10782024-02-15T07:04:09.372ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
10792024-02-15T07:04:09.372ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10802024-02-15T07:04:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10812024-02-15T07:04:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
10822024-02-15T07:04:09.373ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10832024-02-15T07:04:09.373ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
10842024-02-15T07:04:09.373ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
10852024-02-15T07:04:09.373ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
10862024-02-15T07:04:09.373ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 6.57\u{b5}s iteration = 18
10872024-02-15T07:04:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10882024-02-15T07:04:09.373ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10892024-02-15T07:04:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
10902024-02-15T07:04:09.373ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
10912024-02-15T07:04:09.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10922024-02-15T07:04:09.373ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
10932024-02-15T07:04:09.374ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
10942024-02-15T07:04:09.374ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
10952024-02-15T07:04:09.374ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10962024-02-15T07:04:09.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10972024-02-15T07:04:09.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
10982024-02-15T07:04:09.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10992024-02-15T07:04:09.374ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11002024-02-15T07:04:09.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
11012024-02-15T07:04:09.375ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11022024-02-15T07:04:09.375ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
11032024-02-15T07:04:09.375ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = d7f8cbe9-558e-41ea-b105-dcc67f70cc11 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
11042024-02-15T07:04:09.375ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11052024-02-15T07:04:09.375ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11062024-02-15T07:04:09.376ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11072024-02-15T07:04:09.376ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 11.321634ms iteration = 50
11082024-02-15T07:04:09.377ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11092024-02-15T07:04:09.379ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
11102024-02-15T07:04:09.379ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
11112024-02-15T07:04:09.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
11122024-02-15T07:04:09.379ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
11132024-02-15T07:04:09.379ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11142024-02-15T07:04:09.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
11152024-02-15T07:04:09.380ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
11162024-02-15T07:04:09.380ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = c0170a00-b2b4-4879-8162-cfda0a5009a3 dns_zone_name = oxide.test found_so_far = 0 version = 2
11172024-02-15T07:04:09.380ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
11182024-02-15T07:04:09.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
11192024-02-15T07:04:09.381ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11202024-02-15T07:04:09.381ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
11212024-02-15T07:04:09.381ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11222024-02-15T07:04:09.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
11232024-02-15T07:04:09.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11242024-02-15T07:04:09.382ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11252024-02-15T07:04:09.382ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11262024-02-15T07:04:09.383ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = d7f8cbe9-558e-41ea-b105-dcc67f70cc11 dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
11272024-02-15T07:04:09.383ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
11282024-02-15T07:04:09.383ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-02-15 06:56:35.135865 UTC dns_group = internal generation = 1
11292024-02-15T07:04:09.383ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 25.901636ms iteration = 10
11302024-02-15T07:04:09.385ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11312024-02-15T07:04:09.385ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
11322024-02-15T07:04:09.385ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11332024-02-15T07:04:09.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
11342024-02-15T07:04:09.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11352024-02-15T07:04:09.385ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
11362024-02-15T07:04:09.385ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
11372024-02-15T07:04:09.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
11382024-02-15T07:04:09.386ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11392024-02-15T07:04:09.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11402024-02-15T07:04:09.387ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = c0170a00-b2b4-4879-8162-cfda0a5009a3 dns_zone_name = oxide.test found_so_far = 1 version = 2
11412024-02-15T07:04:09.387ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
11422024-02-15T07:04:09.387ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-02-15 06:56:35.417009 UTC dns_group = external generation = 2
11432024-02-15T07:04:09.387ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 22.600016ms iteration = 10
11442024-02-15T07:04:09.388ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11452024-02-15T07:04:09.390ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ec8d62a8-6b82-4888-b333-60568d39e395, lookup_type: ById(ec8d62a8-6b82-4888-b333-60568d39e395) } result = Ok(())
11462024-02-15T07:04:09.390ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11472024-02-15T07:04:09.390ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-02-15T06:56:09.420632Z, time_modified: 2024-02-15T06:56:09.420632Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ffb05f40-4544-4250-8e84-31ffead32a4d, db_silo: Silo { identity: SiloIdentity { id: ffb05f40-4544-4250-8e84-31ffead32a4d, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T06:56:35.393136Z, time_modified: 2024-02-15T06:56:35.393136Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "bc23126eba6ce80ab4599432cb1e3814af0b1a2121110ea710e2bb25f380b4e9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ffb05f40-4544-4250-8e84-31ffead32a4d, db_silo: Silo { identity: SiloIdentity { id: ffb05f40-4544-4250-8e84-31ffead32a4d, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T06:56:35.393136Z, time_modified: 2024-02-15T06:56:35.393136Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "bc23126eba6ce80ab4599432cb1e3814af0b1a2121110ea710e2bb25f380b4e9" }] }) }
11482024-02-15T07:04:09.390ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 26.0651ms iteration = 10
11492024-02-15T07:04:09.390ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11502024-02-15T07:04:09.391ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 27.065482ms iteration = 18
11512024-02-15T07:04:09.427ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
11522024-02-15T07:04:09.430ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
11532024-02-15T07:04:09.490ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
11542024-02-15T07:04:09.490ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
11552024-02-15T07:04:09.491ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "549a6901-dee4-4bdb-b3ee-b449e2563fad", "date": "Thu, 15 Feb 2024 07:04:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
11562024-02-15T07:04:09.491ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
11572024-02-15T07:04:09.492ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "a5d65aff-a1e9-4c1d-8d7f-698fcef72ea9", "date": "Thu, 15 Feb 2024 07:04:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
11582024-02-15T07:04:09.492ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
11592024-02-15T07:04:09.493ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
11602024-02-15T07:04:09.493ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 129.311827ms iteration = 12
11612024-02-15T07:04:09.521ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
11622024-02-15T07:04:09.521ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "133cffa8-0d0c-4f25-8692-1f9ef09c90c7", "date": "Thu, 15 Feb 2024 07:04:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
11632024-02-15T07:04:09.521ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
11642024-02-15T07:04:09.523ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "9ca03d78-928d-45ee-bdc8-33fa4d4d394b", "date": "Thu, 15 Feb 2024 07:04:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
11652024-02-15T07:04:09.523ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
11662024-02-15T07:04:09.541ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 409, headers: {"content-type": "application/json", "x-request-id": "ffcade3d-e42a-4c54-a668-251fe8d2712d", "content-length": "276", "date": "Thu, 15 Feb 2024 07:04:09 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
11672024-02-15T07:04:09.551ZWARNnexus (BackgroundTasks): failed to propagate DNS generation 1 to server [fd00:1122:3344:2::1]:5353: Error Response: status: 409 Conflict; headers: {"content-type": "application/json", "x-request-id": "ffcade3d-e42a-4c54-a668-251fe8d2712d", "content-length": "276", "date": "Thu, 15 Feb 2024 07:04:09 GMT"}; value: Error { error_code: Some("UpdateInProgress"), message: "update already in progress (from req_id \"fe271e1c-17b1-4d36-98c6-663bdebbb5c8\", to generation 1, started at 2024-02-15 07:04:09.515473867 UTC (PT0.006174153S ago))", request_id: "ffcade3d-e42a-4c54-a668-251fe8d2712d" } background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:165 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
11682024-02-15T07:04:09.553ZWARNnexus (BackgroundTasks): DNS propagation: failed background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:104 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
11692024-02-15T07:04:09.553ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 188.535459ms iteration = 12
11702024-02-15T07:04:09.683ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 703 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = d6553898-4484-40c4-9e59-5fcc3deccb8d response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
11712024-02-15T07:04:09.730ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 4bb7769b-e7de-4e32-b120-4e8a61ec2d4a roles = RoleSet { roles: {} } uri = /nat/ipv4/changeset/15?limit=100
11722024-02-15T07:04:09.730ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 4bb7769b-e7de-4e32-b120-4e8a61ec2d4a resource = Database result = Ok(()) uri = /nat/ipv4/changeset/15?limit=100
11732024-02-15T07:04:09.732ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 2872 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::2]:59206 req_id = 4bb7769b-e7de-4e32-b120-4e8a61ec2d4a response_code = 200 uri = /nat/ipv4/changeset/15?limit=100
11742024-02-15T07:04:19.364ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
11752024-02-15T07:04:19.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11762024-02-15T07:04:19.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11772024-02-15T07:04:19.366ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11782024-02-15T07:04:19.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
11792024-02-15T07:04:19.369ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11802024-02-15T07:04:19.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11812024-02-15T07:04:19.370ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.896756ms iteration = 51
11822024-02-15T07:04:19.684ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 987 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = e1792e60-dbfa-4790-947c-923a74906800 response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9
11832024-02-15T07:04:29.365ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 52 reason = Timeout
11842024-02-15T07:04:29.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11852024-02-15T07:04:29.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11862024-02-15T07:04:29.366ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11872024-02-15T07:04:29.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
11882024-02-15T07:04:29.369ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11892024-02-15T07:04:29.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11902024-02-15T07:04:29.371ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.747463ms iteration = 52
11912024-02-15T07:04:29.683ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 539 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44093 req_id = c1b9df71-7858-43dd-98a2-a101d966597d response_code = 200 uri = /metrics/collect/ba8e18ad-524d-4a42-b182-bc303dcab0d9