Max level shown:
12024-04-03T05:15:01.617ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 19 reason = Timeout
22024-04-03T05:15:01.617ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32024-04-03T05:15:01.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42024-04-03T05:15:01.619ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52024-04-03T05:15:01.622ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
62024-04-03T05:15:01.622ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
72024-04-03T05:15:01.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82024-04-03T05:15:01.636ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92024-04-03T05:15:01.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
102024-04-03T05:15:01.638ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112024-04-03T05:15:01.641ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
122024-04-03T05:15:01.641ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
132024-04-03T05:15:01.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
142024-04-03T05:15:01.645ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
152024-04-03T05:15:01.645ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 28.597472ms iteration = 19
162024-04-03T05:15:02.365ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 143 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = f3bfe0fa-3dc5-4175-82e8-ca5668bedd02 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
172024-04-03T05:15:11.616ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 20 reason = Timeout
182024-04-03T05:15:11.616ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 5 reason = Timeout
192024-04-03T05:15:11.616ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
202024-04-03T05:15:11.616ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 5 reason = Timeout
212024-04-03T05:15:11.616ZWARNnexus (BackgroundTasks): Metric producer GC: statically disabled pending omicron#5284 background_task = metrics_producer_gc file = nexus/src/app/background/metrics_producer_gc.rs:43
222024-04-03T05:15:11.616ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 7.5\u{b5}s iteration = 5
232024-04-03T05:15:11.616ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
242024-04-03T05:15:11.616ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 5 reason = Timeout
252024-04-03T05:15:11.617ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
262024-04-03T05:15:11.617ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
272024-04-03T05:15:11.617ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 5 reason = Timeout
282024-04-03T05:15:11.617ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 5 reason = Timeout
292024-04-03T05:15:11.617ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
302024-04-03T05:15:11.617ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
312024-04-03T05:15:11.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
322024-04-03T05:15:11.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
332024-04-03T05:15:11.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
342024-04-03T05:15:11.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
352024-04-03T05:15:11.617ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("174b6905-67c4-445e-8ee0-92f9d656bfce.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1b4f3db-685d-4e05-94cb-90ed84711941.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a2b50f43-9fdf-42d4-88bc-53143a838677.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("a2b50f43-9fdf-42d4-88bc-53143a838677.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 3304, tv_nsec: 301588212 } })
362024-04-03T05:15:11.617ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
372024-04-03T05:15:11.618ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
382024-04-03T05:15:11.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
392024-04-03T05:15:11.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
402024-04-03T05:15:11.618ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
412024-04-03T05:15:11.619ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
422024-04-03T05:15:11.619ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 2.139188ms iteration = 5
432024-04-03T05:15:11.619ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
442024-04-03T05:15:11.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
452024-04-03T05:15:11.620ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
462024-04-03T05:15:11.621ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
472024-04-03T05:15:11.621ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
482024-04-03T05:15:11.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
492024-04-03T05:15:11.623ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
502024-04-03T05:15:11.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
512024-04-03T05:15:11.624ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
522024-04-03T05:15:11.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
532024-04-03T05:15:11.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
542024-04-03T05:15:11.624ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
552024-04-03T05:15:11.624ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
562024-04-03T05:15:11.625ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
572024-04-03T05:15:11.625ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
582024-04-03T05:15:11.625ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
592024-04-03T05:15:11.625ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
602024-04-03T05:15:11.625ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
612024-04-03T05:15:11.625ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
622024-04-03T05:15:11.625ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
632024-04-03T05:15:11.625ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
642024-04-03T05:15:11.625ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
652024-04-03T05:15:11.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
662024-04-03T05:15:11.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
672024-04-03T05:15:11.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
682024-04-03T05:15:11.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
692024-04-03T05:15:11.626ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 6 reason = Timeout
702024-04-03T05:15:11.626ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:65 reason = blueprint disabled target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e
712024-04-03T05:15:11.626ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 16.08\u{b5}s iteration = 6
722024-04-03T05:15:11.627ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
732024-04-03T05:15:11.627ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
742024-04-03T05:15:11.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
752024-04-03T05:15:11.628ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 8 reason = Timeout
762024-04-03T05:15:11.628ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
772024-04-03T05:15:11.628ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
782024-04-03T05:15:11.628ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 8.15\u{b5}s iteration = 8
792024-04-03T05:15:11.628ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
802024-04-03T05:15:11.628ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
812024-04-03T05:15:11.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
822024-04-03T05:15:11.630ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
832024-04-03T05:15:11.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
842024-04-03T05:15:11.631ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
852024-04-03T05:15:11.631ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
862024-04-03T05:15:11.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
872024-04-03T05:15:11.631ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
882024-04-03T05:15:11.631ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
892024-04-03T05:15:11.631ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
902024-04-03T05:15:11.631ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
912024-04-03T05:15:11.631ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
922024-04-03T05:15:11.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
932024-04-03T05:15:11.632ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 8 reason = Timeout
942024-04-03T05:15:11.632ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
952024-04-03T05:15:11.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
962024-04-03T05:15:11.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
972024-04-03T05:15:11.633ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
982024-04-03T05:15:11.633ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 1c2282ea-206d-4df3-bfca-a35a264cd464 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
992024-04-03T05:15:11.633ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1002024-04-03T05:15:11.633ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1012024-04-03T05:15:11.633ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
1022024-04-03T05:15:11.633ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 5844cfd7-028e-4204-ba41-2d4da75c3d1c dns_zone_name = oxide.test found_so_far = 0 version = 2
1032024-04-03T05:15:11.633ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1042024-04-03T05:15:11.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1052024-04-03T05:15:11.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1062024-04-03T05:15:11.633ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
1072024-04-03T05:15:11.633ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1082024-04-03T05:15:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1092024-04-03T05:15:11.634ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1102024-04-03T05:15:11.634ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1112024-04-03T05:15:11.635ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
1122024-04-03T05:15:11.635ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.828951ms iteration = 20
1132024-04-03T05:15:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
1142024-04-03T05:15:11.636ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1152024-04-03T05:15:11.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1162024-04-03T05:15:11.636ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-04-03T05:12:11.760059Z, time_modified: 2024-04-03T05:12:11.760059Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: d254228d-6b17-46e4-bc06-e18669dfc21c, db_silo: Silo { identity: SiloIdentity { id: d254228d-6b17-46e4-bc06-e18669dfc21c, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-04-03T05:12:15.962604Z, time_modified: 2024-04-03T05:12:15.962604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "096b197d9f10cd1b196c898f82b2d3451036c6b5bc8fff5b9af57c633900dba5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: d254228d-6b17-46e4-bc06-e18669dfc21c, db_silo: Silo { identity: SiloIdentity { id: d254228d-6b17-46e4-bc06-e18669dfc21c, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-04-03T05:12:15.962604Z, time_modified: 2024-04-03T05:12:15.962604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "096b197d9f10cd1b196c898f82b2d3451036c6b5bc8fff5b9af57c633900dba5" }] }) }
1172024-04-03T05:15:11.636ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 19.821773ms iteration = 5
1182024-04-03T05:15:11.638ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3304, tv_nsec: 322319118 } })
1192024-04-03T05:15:11.638ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
1202024-04-03T05:15:11.638ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1212024-04-03T05:15:11.638ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
1222024-04-03T05:15:11.638ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1232024-04-03T05:15:11.638ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:974 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
1242024-04-03T05:15:11.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1252024-04-03T05:15:11.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1262024-04-03T05:15:11.640ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 5844cfd7-028e-4204-ba41-2d4da75c3d1c dns_zone_name = oxide.test found_so_far = 1 version = 2
1272024-04-03T05:15:11.640ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
1282024-04-03T05:15:11.640ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-04-03 05:12:15.995755 UTC dns_group = external generation = 2
1292024-04-03T05:15:11.640ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 23.156038ms iteration = 5
1302024-04-03T05:15:11.640ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 1c2282ea-206d-4df3-bfca-a35a264cd464 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
1312024-04-03T05:15:11.640ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
1322024-04-03T05:15:11.640ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-04-03 05:12:15.091055 UTC dns_group = internal generation = 1
1332024-04-03T05:15:11.640ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 23.411842ms iteration = 5
1342024-04-03T05:15:11.641ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 5 reason = Timeout
1352024-04-03T05:15:11.642ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("9f24d5b2-898d-40ee-a459-c6576fb9d418.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c33ab12a-21de-454f-9b76-9a78bfc5c720.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c33ab12a-21de-454f-9b76-9a78bfc5c720.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3304, tv_nsec: 326389142 } })
1362024-04-03T05:15:11.643ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
1372024-04-03T05:15:11.643ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 2.009137ms iteration = 5
1382024-04-03T05:15:11.663ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 8 reason = Timeout
1392024-04-03T05:15:11.663ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
1402024-04-03T05:15:11.663ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 8 reason = Timeout
1412024-04-03T05:15:11.663ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1422024-04-03T05:15:11.663ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 8 reason = Timeout
1432024-04-03T05:15:11.663ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
1442024-04-03T05:15:11.663ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 8 reason = Timeout
1452024-04-03T05:15:11.663ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
1462024-04-03T05:15:11.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1472024-04-03T05:15:11.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1482024-04-03T05:15:11.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
1492024-04-03T05:15:11.664ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1502024-04-03T05:15:11.665ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3304, tv_nsec: 349702853 } })
1512024-04-03T05:15:11.665ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3304, tv_nsec: 349824324 } })
1522024-04-03T05:15:11.666ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:974
1532024-04-03T05:15:11.666ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
1542024-04-03T05:15:11.666ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.078341ms iteration = 8
1552024-04-03T05:15:11.666ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:974
1562024-04-03T05:15:11.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
1572024-04-03T05:15:11.667ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1582024-04-03T05:15:11.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1592024-04-03T05:15:11.668ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1602024-04-03T05:15:11.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1612024-04-03T05:15:11.671ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:982 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 zone_address = fd00:1122:3344:101::2
1622024-04-03T05:15:11.671ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1632024-04-03T05:15:11.671ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "6f60cfff-6ffa-46ad-9d8a-435704696828", "content-length": "26", "date": "Wed, 03 Apr 2024 05:15:11 GMT"} })
1642024-04-03T05:15:11.671ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:987 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1652024-04-03T05:15:11.673ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
1662024-04-03T05:15:11.684ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1672024-04-03T05:15:11.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1682024-04-03T05:15:11.686ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1692024-04-03T05:15:11.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1702024-04-03T05:15:11.687ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1712024-04-03T05:15:11.690ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 54b5d8ce-2f89-4588-80f9-2a71dfa53de1, lookup_type: ById(54b5d8ce-2f89-4588-80f9-2a71dfa53de1) } result = Ok(())
1722024-04-03T05:15:11.690ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1732024-04-03T05:15:11.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1742024-04-03T05:15:11.692ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 28.843204ms iteration = 8
1752024-04-03T05:15:11.758ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
1762024-04-03T05:15:11.758ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1772024-04-03T05:15:11.758ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "63b99e65-68e2-4ab2-ac4c-1b85d407412b", "content-length": "26", "date": "Wed, 03 Apr 2024 05:15:11 GMT"} })
1782024-04-03T05:15:11.758ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1792024-04-03T05:15:11.759ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
1802024-04-03T05:15:11.759ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1812024-04-03T05:15:11.760ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "889e3563-045b-4eb7-959e-c9f8867be416", "content-length": "26", "date": "Wed, 03 Apr 2024 05:15:11 GMT"} })
1822024-04-03T05:15:11.760ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1832024-04-03T05:15:11.760ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1842024-04-03T05:15:11.761ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1852024-04-03T05:15:11.858ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
1862024-04-03T05:15:11.858ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "99101bab-7fd9-452b-a1d8-475eb7773abc", "content-length": "1", "date": "Wed, 03 Apr 2024 05:15:11 GMT"} })
1872024-04-03T05:15:11.860ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
1882024-04-03T05:15:11.860ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1892024-04-03T05:15:11.861ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 198.695907ms iteration = 8
1902024-04-03T05:15:11.936ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 272.826285ms iteration = 8
1912024-04-03T05:15:11.947ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 7 reason = Timeout
1922024-04-03T05:15:11.979ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1932024-04-03T05:15:11.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1942024-04-03T05:15:11.981ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:141 port = SwitchPort { id: 9abb0c53-3a74-441c-8a11-a2b0939760e0, rack_id: 38b3f10b-8962-43bd-8d32-8172eeecefb9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(0a16a58f-d519-4d96-87f3-b4294bb539f8) } rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
1952024-04-03T05:15:11.981ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1962024-04-03T05:15:11.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1972024-04-03T05:15:11.989ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
1982024-04-03T05:15:11.989ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "23dc1b0e-a0f9-462e-ba13-2332d8270956", "content-length": "121", "date": "Wed, 03 Apr 2024 05:15:11 GMT"} })
1992024-04-03T05:15:11.989ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1418 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
2002024-04-03T05:15:11.989ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1445 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 0a16a58f-d519-4d96-87f3-b4294bb539f8
2012024-04-03T05:15:11.990ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:364 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
2022024-04-03T05:15:11.990ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:368 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
2032024-04-03T05:15:11.990ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:377 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
2042024-04-03T05:15:11.990ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:383 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
2052024-04-03T05:15:11.990ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:403 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
2062024-04-03T05:15:11.990ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2072024-04-03T05:15:11.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2082024-04-03T05:15:11.991ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
2092024-04-03T05:15:11.991ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "64285890-7617-4318-8a2c-26a022c44d2b", "content-length": "2", "date": "Wed, 03 Apr 2024 05:15:11 GMT"} })
2102024-04-03T05:15:11.991ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
2112024-04-03T05:15:11.991ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "5c9b02e3-e0c5-464a-9842-42de2e62f555", "content-length": "52", "date": "Wed, 03 Apr 2024 05:15:11 GMT"} })
2122024-04-03T05:15:11.991ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:452 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
2132024-04-03T05:15:11.991ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
2142024-04-03T05:15:11.993ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
2152024-04-03T05:15:12.023ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
2162024-04-03T05:15:12.023ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "dda8f56c-553d-4f26-b92a-569191f23687", "date": "Wed, 03 Apr 2024 05:15:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
2172024-04-03T05:15:12.023ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
2182024-04-03T05:15:12.025ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "13989968-74bd-443b-beb4-83b97f06c5de", "date": "Wed, 03 Apr 2024 05:15:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
2192024-04-03T05:15:12.025ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
2202024-04-03T05:15:12.026ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
2212024-04-03T05:15:12.027ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 79.793223ms iteration = 7
2222024-04-03T05:15:12.041ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "3eb817c6-2ac4-437e-889a-254e4c658c0b", "date": "Wed, 03 Apr 2024 05:15:11 GMT"} })
2232024-04-03T05:15:12.041ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:617 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch = Switch0
2242024-04-03T05:15:12.041ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
2252024-04-03T05:15:12.041ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f6883366-4fe3-4eeb-a261-86b2782e1c45", "content-length": "452", "date": "Wed, 03 Apr 2024 05:15:11 GMT"} })
2262024-04-03T05:15:12.041ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2272024-04-03T05:15:12.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2282024-04-03T05:15:12.047ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2292024-04-03T05:15:12.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2302024-04-03T05:15:12.050ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2312024-04-03T05:15:12.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2322024-04-03T05:15:12.052ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2332024-04-03T05:15:12.052ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2342024-04-03T05:15:12.059ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 427.340882ms iteration = 8
2352024-04-03T05:15:12.290ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 7 reason = Timeout
2362024-04-03T05:15:12.320ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
2372024-04-03T05:15:12.349ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
2382024-04-03T05:15:12.382ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
2392024-04-03T05:15:12.382ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 166 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = c5040c04-af6b-4d29-9a23-d5cacb22609d response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
2402024-04-03T05:15:12.383ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "5f45402c-6a87-4032-b602-d93f245fb3ae", "date": "Wed, 03 Apr 2024 05:15:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
2412024-04-03T05:15:12.383ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
2422024-04-03T05:15:12.385ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "f203619b-ff4e-44c1-ab20-c3170dd4c6b0", "date": "Wed, 03 Apr 2024 05:15:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
2432024-04-03T05:15:12.385ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
2442024-04-03T05:15:12.387ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "2cc04454-4e8c-4e8d-89d4-ede0fd5a4eee", "date": "Wed, 03 Apr 2024 05:15:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
2452024-04-03T05:15:12.387ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
2462024-04-03T05:15:12.389ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
2472024-04-03T05:15:12.389ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 99.201462ms iteration = 7
2482024-04-03T05:15:21.617ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 21 reason = Timeout
2492024-04-03T05:15:21.617ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2502024-04-03T05:15:21.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2512024-04-03T05:15:21.618ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2522024-04-03T05:15:21.621ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2532024-04-03T05:15:21.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2542024-04-03T05:15:21.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2552024-04-03T05:15:21.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2562024-04-03T05:15:21.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2572024-04-03T05:15:21.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2582024-04-03T05:15:21.627ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
2592024-04-03T05:15:21.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2602024-04-03T05:15:21.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2612024-04-03T05:15:21.631ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
2622024-04-03T05:15:21.631ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 14.079797ms iteration = 21
2632024-04-03T05:15:22.366ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 734 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 2044bcd5-dfbc-4b71-b479-ee3dd54d011a response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
2642024-04-03T05:15:31.617ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 22 reason = Timeout
2652024-04-03T05:15:31.617ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2662024-04-03T05:15:31.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2672024-04-03T05:15:31.618ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2682024-04-03T05:15:31.621ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2692024-04-03T05:15:31.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2702024-04-03T05:15:31.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2712024-04-03T05:15:31.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2722024-04-03T05:15:31.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2732024-04-03T05:15:31.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2742024-04-03T05:15:31.626ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
2752024-04-03T05:15:31.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2762024-04-03T05:15:31.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2772024-04-03T05:15:31.630ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
2782024-04-03T05:15:31.630ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 13.429429ms iteration = 22
2792024-04-03T05:15:32.366ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 144 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = a44bce4f-9ab1-488d-976d-10e75d8cef82 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
2802024-04-03T05:15:41.617ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 23 reason = Timeout
2812024-04-03T05:15:41.617ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2822024-04-03T05:15:41.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2832024-04-03T05:15:41.619ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2842024-04-03T05:15:41.621ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2852024-04-03T05:15:41.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2862024-04-03T05:15:41.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2872024-04-03T05:15:41.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2882024-04-03T05:15:41.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2892024-04-03T05:15:41.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2902024-04-03T05:15:41.627ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
2912024-04-03T05:15:41.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2922024-04-03T05:15:41.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2932024-04-03T05:15:41.629ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 9 reason = Timeout
2942024-04-03T05:15:41.629ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
2952024-04-03T05:15:41.629ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
2962024-04-03T05:15:41.629ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 14.21\u{b5}s iteration = 9
2972024-04-03T05:15:41.631ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
2982024-04-03T05:15:41.631ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 14.229249ms iteration = 23
2992024-04-03T05:15:41.633ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 9 reason = Timeout
3002024-04-03T05:15:41.633ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3012024-04-03T05:15:41.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3022024-04-03T05:15:41.634ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3032024-04-03T05:15:41.636ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
3042024-04-03T05:15:41.636ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3052024-04-03T05:15:41.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3062024-04-03T05:15:41.638ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3334, tv_nsec: 322350028 } })
3072024-04-03T05:15:41.639ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:974 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
3082024-04-03T05:15:41.668ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:982 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 zone_address = fd00:1122:3344:101::2
3092024-04-03T05:15:41.668ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
3102024-04-03T05:15:41.669ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 9 reason = Timeout
3112024-04-03T05:15:41.669ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
3122024-04-03T05:15:41.669ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 9 reason = Timeout
3132024-04-03T05:15:41.669ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
3142024-04-03T05:15:41.669ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 9 reason = Timeout
3152024-04-03T05:15:41.669ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 9 reason = Timeout
3162024-04-03T05:15:41.669ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3172024-04-03T05:15:41.669ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3182024-04-03T05:15:41.669ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "0d1b13da-481a-4306-88d5-0780121117e6", "content-length": "26", "date": "Wed, 03 Apr 2024 05:15:41 GMT"} })
3192024-04-03T05:15:41.669ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:987 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
3202024-04-03T05:15:41.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
3212024-04-03T05:15:41.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3222024-04-03T05:15:41.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3232024-04-03T05:15:41.670ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
3242024-04-03T05:15:41.670ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3252024-04-03T05:15:41.671ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3334, tv_nsec: 354900372 } })
3262024-04-03T05:15:41.671ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:974
3272024-04-03T05:15:41.672ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
3282024-04-03T05:15:41.672ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.283043ms iteration = 9
3292024-04-03T05:15:41.672ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3334, tv_nsec: 355992577 } })
3302024-04-03T05:15:41.672ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:974
3312024-04-03T05:15:41.673ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
3322024-04-03T05:15:41.673ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3332024-04-03T05:15:41.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3342024-04-03T05:15:41.674ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3352024-04-03T05:15:41.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3362024-04-03T05:15:41.691ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3372024-04-03T05:15:41.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3382024-04-03T05:15:41.692ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3392024-04-03T05:15:41.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3402024-04-03T05:15:41.693ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3412024-04-03T05:15:41.696ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 54b5d8ce-2f89-4588-80f9-2a71dfa53de1, lookup_type: ById(54b5d8ce-2f89-4588-80f9-2a71dfa53de1) } result = Ok(())
3422024-04-03T05:15:41.696ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3432024-04-03T05:15:41.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3442024-04-03T05:15:41.698ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 29.210609ms iteration = 9
3452024-04-03T05:15:41.773ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
3462024-04-03T05:15:41.773ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
3472024-04-03T05:15:41.774ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2117fd33-85ae-4eb2-82ab-29c1a1f609c2", "content-length": "26", "date": "Wed, 03 Apr 2024 05:15:41 GMT"} })
3482024-04-03T05:15:41.774ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
3492024-04-03T05:15:41.775ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
3502024-04-03T05:15:41.862ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
3512024-04-03T05:15:41.863ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
3522024-04-03T05:15:41.863ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "c661d138-adc5-40a3-bd20-5e623a858984", "content-length": "26", "date": "Wed, 03 Apr 2024 05:15:41 GMT"} })
3532024-04-03T05:15:41.863ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
3542024-04-03T05:15:41.865ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
3552024-04-03T05:15:41.865ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 196.143062ms iteration = 9
3562024-04-03T05:15:41.901ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
3572024-04-03T05:15:41.902ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f9cb3509-7011-4ca1-83d4-42eef1496d47", "content-length": "1", "date": "Wed, 03 Apr 2024 05:15:41 GMT"} })
3582024-04-03T05:15:41.904ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3592024-04-03T05:15:41.905ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3602024-04-03T05:15:41.906ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 237.542435ms iteration = 9
3612024-04-03T05:15:41.928ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3622024-04-03T05:15:41.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3632024-04-03T05:15:41.930ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:141 port = SwitchPort { id: 9abb0c53-3a74-441c-8a11-a2b0939760e0, rack_id: 38b3f10b-8962-43bd-8d32-8172eeecefb9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(0a16a58f-d519-4d96-87f3-b4294bb539f8) } rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
3642024-04-03T05:15:41.930ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3652024-04-03T05:15:41.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3662024-04-03T05:15:41.938ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
3672024-04-03T05:15:41.939ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "560454ef-6daa-42ca-a3df-4ca438778f79", "content-length": "121", "date": "Wed, 03 Apr 2024 05:15:41 GMT"} })
3682024-04-03T05:15:41.939ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1418 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
3692024-04-03T05:15:41.939ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1445 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 0a16a58f-d519-4d96-87f3-b4294bb539f8
3702024-04-03T05:15:41.939ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:364 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
3712024-04-03T05:15:41.939ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:368 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
3722024-04-03T05:15:41.939ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:377 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
3732024-04-03T05:15:41.939ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:383 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
3742024-04-03T05:15:41.939ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:403 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
3752024-04-03T05:15:41.939ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3762024-04-03T05:15:41.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3772024-04-03T05:15:41.940ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
3782024-04-03T05:15:41.940ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "95930027-7451-4d03-8c4e-7768db9568a7", "content-length": "2", "date": "Wed, 03 Apr 2024 05:15:41 GMT"} })
3792024-04-03T05:15:41.940ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
3802024-04-03T05:15:41.940ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f31213b1-c05d-47c5-98a8-fa3b41c4d6ce", "content-length": "52", "date": "Wed, 03 Apr 2024 05:15:41 GMT"} })
3812024-04-03T05:15:41.940ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:452 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
3822024-04-03T05:15:41.940ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
3832024-04-03T05:15:41.991ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "a297c653-40ed-4a7c-8421-c12d52f77cb2", "date": "Wed, 03 Apr 2024 05:15:41 GMT"} })
3842024-04-03T05:15:41.991ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:617 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch = Switch0
3852024-04-03T05:15:41.991ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
3862024-04-03T05:15:41.994ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "c178f23f-31e0-42f3-8957-224c97fde2c9", "content-length": "452", "date": "Wed, 03 Apr 2024 05:15:41 GMT"} })
3872024-04-03T05:15:41.994ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3882024-04-03T05:15:41.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3892024-04-03T05:15:41.998ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3902024-04-03T05:15:41.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3912024-04-03T05:15:42.001ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3922024-04-03T05:15:42.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3932024-04-03T05:15:42.002ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3942024-04-03T05:15:42.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3952024-04-03T05:15:42.008ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 374.830314ms iteration = 9
3962024-04-03T05:15:42.366ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 143 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 9860603d-a064-4d42-bff9-eecc643a3003 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
3972024-04-03T05:15:51.616ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 24 reason = Timeout
3982024-04-03T05:15:51.616ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3992024-04-03T05:15:51.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4002024-04-03T05:15:51.618ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4012024-04-03T05:15:51.621ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4022024-04-03T05:15:51.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4032024-04-03T05:15:51.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4042024-04-03T05:15:51.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4052024-04-03T05:15:51.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4062024-04-03T05:15:51.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4072024-04-03T05:15:51.627ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
4082024-04-03T05:15:51.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4092024-04-03T05:15:51.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4102024-04-03T05:15:51.631ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
4112024-04-03T05:15:51.631ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 14.331511ms iteration = 24
4122024-04-03T05:15:52.366ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 147 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 74369e80-851f-42cd-a507-1ee0ddd77d19 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
4132024-04-03T05:16:01.617ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 25 reason = Timeout
4142024-04-03T05:16:01.617ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4152024-04-03T05:16:01.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4162024-04-03T05:16:01.619ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4172024-04-03T05:16:01.622ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4182024-04-03T05:16:01.622ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4192024-04-03T05:16:01.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4202024-04-03T05:16:01.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4212024-04-03T05:16:01.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4222024-04-03T05:16:01.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4232024-04-03T05:16:01.628ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
4242024-04-03T05:16:01.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4252024-04-03T05:16:01.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4262024-04-03T05:16:01.632ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
4272024-04-03T05:16:01.632ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 14.973169ms iteration = 25
4282024-04-03T05:16:02.367ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 740 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 318f86c1-45e0-4549-8089-86f3af5ac4d5 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
4292024-04-03T05:16:11.617ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 26 reason = Timeout
4302024-04-03T05:16:11.617ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 6 reason = Timeout
4312024-04-03T05:16:11.617ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4322024-04-03T05:16:11.617ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 6 reason = Timeout
4332024-04-03T05:16:11.617ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4342024-04-03T05:16:11.617ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
4352024-04-03T05:16:11.617ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 6 reason = Timeout
4362024-04-03T05:16:11.617ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4372024-04-03T05:16:11.617ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
4382024-04-03T05:16:11.617ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 6 reason = Timeout
4392024-04-03T05:16:11.617ZWARNnexus (BackgroundTasks): Metric producer GC: statically disabled pending omicron#5284 background_task = metrics_producer_gc file = nexus/src/app/background/metrics_producer_gc.rs:43
4402024-04-03T05:16:11.617ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4412024-04-03T05:16:11.617ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 7.97\u{b5}s iteration = 6
4422024-04-03T05:16:11.617ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 6 reason = Timeout
4432024-04-03T05:16:11.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4442024-04-03T05:16:11.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4452024-04-03T05:16:11.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4462024-04-03T05:16:11.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4472024-04-03T05:16:11.618ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("174b6905-67c4-445e-8ee0-92f9d656bfce.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1b4f3db-685d-4e05-94cb-90ed84711941.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a2b50f43-9fdf-42d4-88bc-53143a838677.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("a2b50f43-9fdf-42d4-88bc-53143a838677.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 3364, tv_nsec: 301593731 } })
4482024-04-03T05:16:11.619ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4492024-04-03T05:16:11.619ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4502024-04-03T05:16:11.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4512024-04-03T05:16:11.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4522024-04-03T05:16:11.620ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
4532024-04-03T05:16:11.620ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 2.231979ms iteration = 6
4542024-04-03T05:16:11.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4552024-04-03T05:16:11.620ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4562024-04-03T05:16:11.621ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4572024-04-03T05:16:11.621ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4582024-04-03T05:16:11.624ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
4592024-04-03T05:16:11.624ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4602024-04-03T05:16:11.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4612024-04-03T05:16:11.624ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4622024-04-03T05:16:11.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4632024-04-03T05:16:11.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4642024-04-03T05:16:11.625ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
4652024-04-03T05:16:11.625ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
4662024-04-03T05:16:11.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4672024-04-03T05:16:11.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4682024-04-03T05:16:11.626ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4692024-04-03T05:16:11.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4702024-04-03T05:16:11.627ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
4712024-04-03T05:16:11.627ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
4722024-04-03T05:16:11.627ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
4732024-04-03T05:16:11.627ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4742024-04-03T05:16:11.627ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
4752024-04-03T05:16:11.627ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
4762024-04-03T05:16:11.627ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
4772024-04-03T05:16:11.627ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4782024-04-03T05:16:11.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4792024-04-03T05:16:11.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4802024-04-03T05:16:11.627ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 7 reason = Timeout
4812024-04-03T05:16:11.627ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:65 reason = blueprint disabled target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e
4822024-04-03T05:16:11.627ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 9.75\u{b5}s iteration = 7
4832024-04-03T05:16:11.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4842024-04-03T05:16:11.627ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4852024-04-03T05:16:11.628ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4862024-04-03T05:16:11.628ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4872024-04-03T05:16:11.629ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 10 reason = Timeout
4882024-04-03T05:16:11.629ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
4892024-04-03T05:16:11.629ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
4902024-04-03T05:16:11.629ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 7.2\u{b5}s iteration = 10
4912024-04-03T05:16:11.631ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
4922024-04-03T05:16:11.631ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4932024-04-03T05:16:11.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4942024-04-03T05:16:11.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
4952024-04-03T05:16:11.632ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4962024-04-03T05:16:11.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4972024-04-03T05:16:11.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
4982024-04-03T05:16:11.632ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4992024-04-03T05:16:11.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5002024-04-03T05:16:11.632ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5012024-04-03T05:16:11.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5022024-04-03T05:16:11.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5032024-04-03T05:16:11.633ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 10 reason = Timeout
5042024-04-03T05:16:11.633ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5052024-04-03T05:16:11.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5062024-04-03T05:16:11.633ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5072024-04-03T05:16:11.634ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
5082024-04-03T05:16:11.634ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 1c2282ea-206d-4df3-bfca-a35a264cd464 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
5092024-04-03T05:16:11.634ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5102024-04-03T05:16:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5112024-04-03T05:16:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5122024-04-03T05:16:11.634ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
5132024-04-03T05:16:11.634ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 5844cfd7-028e-4204-ba41-2d4da75c3d1c dns_zone_name = oxide.test found_so_far = 0 version = 2
5142024-04-03T05:16:11.634ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5152024-04-03T05:16:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5162024-04-03T05:16:11.634ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5172024-04-03T05:16:11.635ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5182024-04-03T05:16:11.635ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5192024-04-03T05:16:11.635ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5202024-04-03T05:16:11.636ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
5212024-04-03T05:16:11.636ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.376485ms iteration = 26
5222024-04-03T05:16:11.636ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
5232024-04-03T05:16:11.636ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5242024-04-03T05:16:11.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5252024-04-03T05:16:11.638ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
5262024-04-03T05:16:11.638ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5272024-04-03T05:16:11.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5282024-04-03T05:16:11.638ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3364, tv_nsec: 321598617 } })
5292024-04-03T05:16:11.638ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
5302024-04-03T05:16:11.638ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5312024-04-03T05:16:11.639ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:974 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
5322024-04-03T05:16:11.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5332024-04-03T05:16:11.639ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5342024-04-03T05:16:11.639ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5352024-04-03T05:16:11.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5362024-04-03T05:16:11.640ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 5844cfd7-028e-4204-ba41-2d4da75c3d1c dns_zone_name = oxide.test found_so_far = 1 version = 2
5372024-04-03T05:16:11.640ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
5382024-04-03T05:16:11.640ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-04-03 05:12:15.995755 UTC dns_group = external generation = 2
5392024-04-03T05:16:11.640ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 22.987966ms iteration = 6
5402024-04-03T05:16:11.641ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-04-03T05:12:11.760059Z, time_modified: 2024-04-03T05:12:11.760059Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: d254228d-6b17-46e4-bc06-e18669dfc21c, db_silo: Silo { identity: SiloIdentity { id: d254228d-6b17-46e4-bc06-e18669dfc21c, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-04-03T05:12:15.962604Z, time_modified: 2024-04-03T05:12:15.962604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "096b197d9f10cd1b196c898f82b2d3451036c6b5bc8fff5b9af57c633900dba5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: d254228d-6b17-46e4-bc06-e18669dfc21c, db_silo: Silo { identity: SiloIdentity { id: d254228d-6b17-46e4-bc06-e18669dfc21c, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-04-03T05:12:15.962604Z, time_modified: 2024-04-03T05:12:15.962604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "096b197d9f10cd1b196c898f82b2d3451036c6b5bc8fff5b9af57c633900dba5" }] }) }
5412024-04-03T05:16:11.641ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 23.927789ms iteration = 6
5422024-04-03T05:16:11.642ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 6 reason = Timeout
5432024-04-03T05:16:11.643ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("9f24d5b2-898d-40ee-a459-c6576fb9d418.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c33ab12a-21de-454f-9b76-9a78bfc5c720.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c33ab12a-21de-454f-9b76-9a78bfc5c720.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3364, tv_nsec: 326668775 } })
5442024-04-03T05:16:11.644ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
5452024-04-03T05:16:11.644ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.763423ms iteration = 6
5462024-04-03T05:16:11.670ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:982 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 zone_address = fd00:1122:3344:101::2
5472024-04-03T05:16:11.670ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5482024-04-03T05:16:11.670ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 10 reason = Timeout
5492024-04-03T05:16:11.670ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
5502024-04-03T05:16:11.670ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 10 reason = Timeout
5512024-04-03T05:16:11.670ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 10 reason = Timeout
5522024-04-03T05:16:11.670ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
5532024-04-03T05:16:11.670ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 10 reason = Timeout
5542024-04-03T05:16:11.670ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5552024-04-03T05:16:11.670ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
5562024-04-03T05:16:11.670ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "819534d5-1fe0-46ae-8c11-64829418233a", "content-length": "26", "date": "Wed, 03 Apr 2024 05:16:11 GMT"} })
5572024-04-03T05:16:11.670ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:987 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5582024-04-03T05:16:11.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
5592024-04-03T05:16:11.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
5602024-04-03T05:16:11.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5612024-04-03T05:16:11.672ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
5622024-04-03T05:16:11.672ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5632024-04-03T05:16:11.673ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3364, tv_nsec: 356182848 } })
5642024-04-03T05:16:11.673ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:974
5652024-04-03T05:16:11.673ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
5662024-04-03T05:16:11.673ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.328954ms iteration = 10
5672024-04-03T05:16:11.674ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3364, tv_nsec: 357207142 } })
5682024-04-03T05:16:11.674ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:974
5692024-04-03T05:16:11.675ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
5702024-04-03T05:16:11.675ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5712024-04-03T05:16:11.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5722024-04-03T05:16:11.676ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5732024-04-03T05:16:11.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5742024-04-03T05:16:11.693ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5752024-04-03T05:16:11.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5762024-04-03T05:16:11.695ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5772024-04-03T05:16:11.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5782024-04-03T05:16:11.696ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 1c2282ea-206d-4df3-bfca-a35a264cd464 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
5792024-04-03T05:16:11.696ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
5802024-04-03T05:16:11.696ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-04-03 05:12:15.091055 UTC dns_group = internal generation = 1
5812024-04-03T05:16:11.696ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 79.026723ms iteration = 6
5822024-04-03T05:16:11.697ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5832024-04-03T05:16:11.701ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 54b5d8ce-2f89-4588-80f9-2a71dfa53de1, lookup_type: ById(54b5d8ce-2f89-4588-80f9-2a71dfa53de1) } result = Ok(())
5842024-04-03T05:16:11.701ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5852024-04-03T05:16:11.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5862024-04-03T05:16:11.702ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 32.562014ms iteration = 10
5872024-04-03T05:16:11.727ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
5882024-04-03T05:16:11.727ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5892024-04-03T05:16:11.727ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
5902024-04-03T05:16:11.727ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5912024-04-03T05:16:11.728ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "bd082868-dd00-49ad-bb45-56495b4c7084", "content-length": "26", "date": "Wed, 03 Apr 2024 05:16:11 GMT"} })
5922024-04-03T05:16:11.728ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5932024-04-03T05:16:11.728ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "ca9232e5-235a-4d31-a720-7e9c7a931ba1", "content-length": "26", "date": "Wed, 03 Apr 2024 05:16:11 GMT"} })
5942024-04-03T05:16:11.728ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5952024-04-03T05:16:11.730ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
5962024-04-03T05:16:11.730ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
5972024-04-03T05:16:11.804ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
5982024-04-03T05:16:11.805ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "48f2eccc-6598-4e67-8755-7535ac50aa10", "content-length": "1", "date": "Wed, 03 Apr 2024 05:16:11 GMT"} })
5992024-04-03T05:16:11.805ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6002024-04-03T05:16:11.806ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6012024-04-03T05:16:11.807ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 137.231579ms iteration = 10
6022024-04-03T05:16:11.811ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 141.511035ms iteration = 10
6032024-04-03T05:16:11.840ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6042024-04-03T05:16:11.840ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6052024-04-03T05:16:11.842ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:141 port = SwitchPort { id: 9abb0c53-3a74-441c-8a11-a2b0939760e0, rack_id: 38b3f10b-8962-43bd-8d32-8172eeecefb9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(0a16a58f-d519-4d96-87f3-b4294bb539f8) } rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
6062024-04-03T05:16:11.842ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6072024-04-03T05:16:11.842ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6082024-04-03T05:16:11.848ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
6092024-04-03T05:16:11.848ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f64a7bbc-58b1-4c21-bcce-5844afd73e38", "content-length": "121", "date": "Wed, 03 Apr 2024 05:16:11 GMT"} })
6102024-04-03T05:16:11.848ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1418 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
6112024-04-03T05:16:11.848ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1445 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 0a16a58f-d519-4d96-87f3-b4294bb539f8
6122024-04-03T05:16:11.849ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:364 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
6132024-04-03T05:16:11.849ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:368 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
6142024-04-03T05:16:11.849ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:377 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
6152024-04-03T05:16:11.849ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:383 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
6162024-04-03T05:16:11.849ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:403 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
6172024-04-03T05:16:11.849ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6182024-04-03T05:16:11.849ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6192024-04-03T05:16:11.850ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
6202024-04-03T05:16:11.850ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "882279bc-f652-4c9a-8b11-b0a992510c9b", "content-length": "2", "date": "Wed, 03 Apr 2024 05:16:11 GMT"} })
6212024-04-03T05:16:11.850ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
6222024-04-03T05:16:11.850ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "96ec0810-86d7-4624-9579-c03f27cabd31", "content-length": "52", "date": "Wed, 03 Apr 2024 05:16:11 GMT"} })
6232024-04-03T05:16:11.850ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:452 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
6242024-04-03T05:16:11.850ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
6252024-04-03T05:16:11.903ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "ea83c5df-c375-446e-9aaf-86bd87bec686", "date": "Wed, 03 Apr 2024 05:16:11 GMT"} })
6262024-04-03T05:16:11.903ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:617 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch = Switch0
6272024-04-03T05:16:11.903ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
6282024-04-03T05:16:11.904ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e6963c2d-6e28-41c9-be06-64a881517e69", "content-length": "452", "date": "Wed, 03 Apr 2024 05:16:11 GMT"} })
6292024-04-03T05:16:11.904ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6302024-04-03T05:16:11.904ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6312024-04-03T05:16:11.907ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6322024-04-03T05:16:11.907ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6332024-04-03T05:16:11.909ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6342024-04-03T05:16:11.910ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6352024-04-03T05:16:11.911ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6362024-04-03T05:16:11.911ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6372024-04-03T05:16:11.918ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 285.279501ms iteration = 10
6382024-04-03T05:16:11.947ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 8 reason = Timeout
6392024-04-03T05:16:11.977ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
6402024-04-03T05:16:12.016ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
6412024-04-03T05:16:12.016ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "59a4291a-8380-47e2-829b-60cc12f70572", "date": "Wed, 03 Apr 2024 05:16:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
6422024-04-03T05:16:12.016ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
6432024-04-03T05:16:12.017ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "2dd51f17-3b91-4b96-b6a6-cb71a6caa9e2", "date": "Wed, 03 Apr 2024 05:16:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
6442024-04-03T05:16:12.017ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
6452024-04-03T05:16:12.017ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
6462024-04-03T05:16:12.017ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 70.474768ms iteration = 8
6472024-04-03T05:16:12.291ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 8 reason = Timeout
6482024-04-03T05:16:12.320ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
6492024-04-03T05:16:12.350ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
6502024-04-03T05:16:12.380ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
6512024-04-03T05:16:12.381ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 790 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 62023d55-742f-4075-a73d-3f4613f1ce8b response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
6522024-04-03T05:16:12.381ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "d047f0ba-49a7-4d43-821e-975aa13eb75b", "date": "Wed, 03 Apr 2024 05:16:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
6532024-04-03T05:16:12.381ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
6542024-04-03T05:16:12.381ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "f68cac19-f365-4e1a-8903-51702e72b252", "date": "Wed, 03 Apr 2024 05:16:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
6552024-04-03T05:16:12.381ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
6562024-04-03T05:16:12.382ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "278177e2-30d3-4bbe-8dcb-32824958c7ea", "date": "Wed, 03 Apr 2024 05:16:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
6572024-04-03T05:16:12.382ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
6582024-04-03T05:16:12.384ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
6592024-04-03T05:16:12.384ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 92.861397ms iteration = 8
6602024-04-03T05:16:21.617ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 27 reason = Timeout
6612024-04-03T05:16:21.617ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6622024-04-03T05:16:21.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6632024-04-03T05:16:21.619ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6642024-04-03T05:16:21.621ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
6652024-04-03T05:16:21.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6662024-04-03T05:16:21.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6672024-04-03T05:16:21.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6682024-04-03T05:16:21.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6692024-04-03T05:16:21.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6702024-04-03T05:16:21.627ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
6712024-04-03T05:16:21.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6722024-04-03T05:16:21.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6732024-04-03T05:16:21.632ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
6742024-04-03T05:16:21.632ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 14.99865ms iteration = 27
6752024-04-03T05:16:22.367ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 130 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = a2fee755-41b0-4e0b-a8ca-55a7221b3f66 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
6762024-04-03T05:16:31.618ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 28 reason = Timeout
6772024-04-03T05:16:31.618ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6782024-04-03T05:16:31.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6792024-04-03T05:16:31.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6802024-04-03T05:16:31.623ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
6812024-04-03T05:16:31.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6822024-04-03T05:16:31.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6832024-04-03T05:16:31.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6842024-04-03T05:16:31.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6852024-04-03T05:16:31.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6862024-04-03T05:16:31.629ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
6872024-04-03T05:16:31.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6882024-04-03T05:16:31.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6892024-04-03T05:16:31.633ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
6902024-04-03T05:16:31.633ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.315664ms iteration = 28
6912024-04-03T05:16:32.367ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 756 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 9c7b39d7-ba5f-4a28-9c6f-99ece754a26f response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
6922024-04-03T05:16:41.618ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 29 reason = Timeout
6932024-04-03T05:16:41.618ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6942024-04-03T05:16:41.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6952024-04-03T05:16:41.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6962024-04-03T05:16:41.623ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
6972024-04-03T05:16:41.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6982024-04-03T05:16:41.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6992024-04-03T05:16:41.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7002024-04-03T05:16:41.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7012024-04-03T05:16:41.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7022024-04-03T05:16:41.629ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
7032024-04-03T05:16:41.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7042024-04-03T05:16:41.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7052024-04-03T05:16:41.629ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 11 reason = Timeout
7062024-04-03T05:16:41.629ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
7072024-04-03T05:16:41.629ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
7082024-04-03T05:16:41.629ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 9.32\u{b5}s iteration = 11
7092024-04-03T05:16:41.633ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
7102024-04-03T05:16:41.633ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 14.842248ms iteration = 29
7112024-04-03T05:16:41.633ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 11 reason = Timeout
7122024-04-03T05:16:41.633ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7132024-04-03T05:16:41.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7142024-04-03T05:16:41.635ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7152024-04-03T05:16:41.637ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
7162024-04-03T05:16:41.637ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7172024-04-03T05:16:41.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7182024-04-03T05:16:41.639ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3394, tv_nsec: 322001362 } })
7192024-04-03T05:16:41.639ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:974 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
7202024-04-03T05:16:41.669ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:982 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 zone_address = fd00:1122:3344:101::2
7212024-04-03T05:16:41.669ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
7222024-04-03T05:16:41.669ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "0bb5972a-71f9-41c5-9e58-e211ef2dd858", "content-length": "26", "date": "Wed, 03 Apr 2024 05:16:41 GMT"} })
7232024-04-03T05:16:41.669ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:987 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
7242024-04-03T05:16:41.670ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
7252024-04-03T05:16:41.671ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 11 reason = Timeout
7262024-04-03T05:16:41.671ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 11 reason = Timeout
7272024-04-03T05:16:41.671ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
7282024-04-03T05:16:41.671ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7292024-04-03T05:16:41.671ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 11 reason = Timeout
7302024-04-03T05:16:41.671ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7312024-04-03T05:16:41.671ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 11 reason = Timeout
7322024-04-03T05:16:41.671ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
7332024-04-03T05:16:41.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7342024-04-03T05:16:41.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
7352024-04-03T05:16:41.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7362024-04-03T05:16:41.673ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7372024-04-03T05:16:41.673ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3394, tv_nsec: 356360850 } })
7382024-04-03T05:16:41.674ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
7392024-04-03T05:16:41.674ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.465603ms iteration = 11
7402024-04-03T05:16:41.674ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:974
7412024-04-03T05:16:41.676ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
7422024-04-03T05:16:41.676ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7432024-04-03T05:16:41.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7442024-04-03T05:16:41.676ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3394, tv_nsec: 359525332 } })
7452024-04-03T05:16:41.677ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:974
7462024-04-03T05:16:41.677ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7472024-04-03T05:16:41.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7482024-04-03T05:16:41.694ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7492024-04-03T05:16:41.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7502024-04-03T05:16:41.695ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7512024-04-03T05:16:41.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7522024-04-03T05:16:41.697ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7532024-04-03T05:16:41.700ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 54b5d8ce-2f89-4588-80f9-2a71dfa53de1, lookup_type: ById(54b5d8ce-2f89-4588-80f9-2a71dfa53de1) } result = Ok(())
7542024-04-03T05:16:41.700ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7552024-04-03T05:16:41.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7562024-04-03T05:16:41.701ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 30.262283ms iteration = 11
7572024-04-03T05:16:41.766ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
7582024-04-03T05:16:41.766ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
7592024-04-03T05:16:41.766ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "759fc251-6b58-441c-8dd5-09246aef5cbf", "content-length": "26", "date": "Wed, 03 Apr 2024 05:16:41 GMT"} })
7602024-04-03T05:16:41.766ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
7612024-04-03T05:16:41.768ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
7622024-04-03T05:16:41.770ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
7632024-04-03T05:16:41.770ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
7642024-04-03T05:16:41.770ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "73e45426-17da-454d-861e-fa8716af5f6b", "content-length": "26", "date": "Wed, 03 Apr 2024 05:16:41 GMT"} })
7652024-04-03T05:16:41.770ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
7662024-04-03T05:16:41.773ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
7672024-04-03T05:16:42.073ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
7682024-04-03T05:16:42.073ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "a58fb22f-ce24-44da-9e6f-b43e799980ed", "content-length": "1", "date": "Wed, 03 Apr 2024 05:16:41 GMT"} })
7692024-04-03T05:16:42.075ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7702024-04-03T05:16:42.075ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 403.994562ms iteration = 11
7712024-04-03T05:16:42.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7722024-04-03T05:16:42.077ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 405.488792ms iteration = 11
7732024-04-03T05:16:42.098ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7742024-04-03T05:16:42.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7752024-04-03T05:16:42.101ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:141 port = SwitchPort { id: 9abb0c53-3a74-441c-8a11-a2b0939760e0, rack_id: 38b3f10b-8962-43bd-8d32-8172eeecefb9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(0a16a58f-d519-4d96-87f3-b4294bb539f8) } rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
7762024-04-03T05:16:42.101ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7772024-04-03T05:16:42.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7782024-04-03T05:16:42.112ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
7792024-04-03T05:16:42.112ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f95fca42-d3c0-40c8-af5f-562898b5b786", "content-length": "121", "date": "Wed, 03 Apr 2024 05:16:41 GMT"} })
7802024-04-03T05:16:42.112ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1418 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
7812024-04-03T05:16:42.112ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1445 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 0a16a58f-d519-4d96-87f3-b4294bb539f8
7822024-04-03T05:16:42.112ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:364 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
7832024-04-03T05:16:42.112ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:368 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
7842024-04-03T05:16:42.112ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:377 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
7852024-04-03T05:16:42.112ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:383 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
7862024-04-03T05:16:42.112ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:403 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
7872024-04-03T05:16:42.112ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7882024-04-03T05:16:42.113ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7892024-04-03T05:16:42.113ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
7902024-04-03T05:16:42.114ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "5e44bd3c-432d-434f-96b2-27a123e9b845", "content-length": "2", "date": "Wed, 03 Apr 2024 05:16:41 GMT"} })
7912024-04-03T05:16:42.114ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
7922024-04-03T05:16:42.114ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "90c61a0a-a66d-4be9-8d09-17ef34804ef3", "content-length": "52", "date": "Wed, 03 Apr 2024 05:16:41 GMT"} })
7932024-04-03T05:16:42.114ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:452 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
7942024-04-03T05:16:42.114ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
7952024-04-03T05:16:42.161ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "e2018742-f042-45e0-87be-aac7a7d6f3c6", "date": "Wed, 03 Apr 2024 05:16:41 GMT"} })
7962024-04-03T05:16:42.161ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:617 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch = Switch0
7972024-04-03T05:16:42.161ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
7982024-04-03T05:16:42.161ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "be2a799a-643f-426e-a86d-434887e00a8a", "content-length": "452", "date": "Wed, 03 Apr 2024 05:16:41 GMT"} })
7992024-04-03T05:16:42.161ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8002024-04-03T05:16:42.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8012024-04-03T05:16:42.164ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8022024-04-03T05:16:42.164ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8032024-04-03T05:16:42.168ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8042024-04-03T05:16:42.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8052024-04-03T05:16:42.169ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8062024-04-03T05:16:42.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8072024-04-03T05:16:42.175ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 542.011881ms iteration = 11
8082024-04-03T05:16:42.366ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 142 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 9b31d20f-1aaf-43d0-88fa-404941303631 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
8092024-04-03T05:16:51.618ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 30 reason = Timeout
8102024-04-03T05:16:51.618ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8112024-04-03T05:16:51.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8122024-04-03T05:16:51.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8132024-04-03T05:16:51.623ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8142024-04-03T05:16:51.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8152024-04-03T05:16:51.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8162024-04-03T05:16:51.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8172024-04-03T05:16:51.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8182024-04-03T05:16:51.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8192024-04-03T05:16:51.629ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
8202024-04-03T05:16:51.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8212024-04-03T05:16:51.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8222024-04-03T05:16:51.633ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
8232024-04-03T05:16:51.633ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.616528ms iteration = 30
8242024-04-03T05:16:52.367ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 125 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = ad2e9ee6-5e62-42b1-a81c-677ea8689ed8 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
8252024-04-03T05:17:01.618ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 31 reason = Timeout
8262024-04-03T05:17:01.618ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8272024-04-03T05:17:01.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8282024-04-03T05:17:01.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8292024-04-03T05:17:01.623ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8302024-04-03T05:17:01.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8312024-04-03T05:17:01.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8322024-04-03T05:17:01.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8332024-04-03T05:17:01.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8342024-04-03T05:17:01.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8352024-04-03T05:17:01.629ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
8362024-04-03T05:17:01.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8372024-04-03T05:17:01.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8382024-04-03T05:17:01.633ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
8392024-04-03T05:17:01.633ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.354174ms iteration = 31
8402024-04-03T05:17:02.367ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 129 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 2decafee-9fd7-46e5-821f-103b3c9a2dc2 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
8412024-04-03T05:17:11.618ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 32 reason = Timeout
8422024-04-03T05:17:11.618ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 7 reason = Timeout
8432024-04-03T05:17:11.618ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8442024-04-03T05:17:11.618ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
8452024-04-03T05:17:11.618ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 7 reason = Timeout
8462024-04-03T05:17:11.618ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8472024-04-03T05:17:11.618ZWARNnexus (BackgroundTasks): Metric producer GC: statically disabled pending omicron#5284 background_task = metrics_producer_gc file = nexus/src/app/background/metrics_producer_gc.rs:43
8482024-04-03T05:17:11.618ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 8.66\u{b5}s iteration = 7
8492024-04-03T05:17:11.618ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 7 reason = Timeout
8502024-04-03T05:17:11.618ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 7 reason = Timeout
8512024-04-03T05:17:11.619ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8522024-04-03T05:17:11.619ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 7 reason = Timeout
8532024-04-03T05:17:11.619ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
8542024-04-03T05:17:11.619ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8552024-04-03T05:17:11.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8562024-04-03T05:17:11.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8572024-04-03T05:17:11.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8582024-04-03T05:17:11.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8592024-04-03T05:17:11.619ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("174b6905-67c4-445e-8ee0-92f9d656bfce.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1b4f3db-685d-4e05-94cb-90ed84711941.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a2b50f43-9fdf-42d4-88bc-53143a838677.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("a2b50f43-9fdf-42d4-88bc-53143a838677.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 3424, tv_nsec: 301747862 } })
8602024-04-03T05:17:11.620ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8612024-04-03T05:17:11.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8622024-04-03T05:17:11.620ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8632024-04-03T05:17:11.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8642024-04-03T05:17:11.620ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
8652024-04-03T05:17:11.620ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 2.068818ms iteration = 7
8662024-04-03T05:17:11.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8672024-04-03T05:17:11.621ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8682024-04-03T05:17:11.621ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8692024-04-03T05:17:11.622ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8702024-04-03T05:17:11.625ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
8712024-04-03T05:17:11.625ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8722024-04-03T05:17:11.625ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8732024-04-03T05:17:11.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8742024-04-03T05:17:11.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8752024-04-03T05:17:11.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8762024-04-03T05:17:11.626ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
8772024-04-03T05:17:11.626ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
8782024-04-03T05:17:11.627ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8792024-04-03T05:17:11.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8802024-04-03T05:17:11.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8812024-04-03T05:17:11.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8822024-04-03T05:17:11.627ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
8832024-04-03T05:17:11.627ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
8842024-04-03T05:17:11.627ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
8852024-04-03T05:17:11.627ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8862024-04-03T05:17:11.627ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
8872024-04-03T05:17:11.627ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
8882024-04-03T05:17:11.627ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
8892024-04-03T05:17:11.627ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8902024-04-03T05:17:11.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8912024-04-03T05:17:11.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8922024-04-03T05:17:11.628ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 8 reason = Timeout
8932024-04-03T05:17:11.628ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:65 reason = blueprint disabled target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e
8942024-04-03T05:17:11.628ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.49\u{b5}s iteration = 8
8952024-04-03T05:17:11.628ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8962024-04-03T05:17:11.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8972024-04-03T05:17:11.629ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8982024-04-03T05:17:11.629ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8992024-04-03T05:17:11.630ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 12 reason = Timeout
9002024-04-03T05:17:11.630ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
9012024-04-03T05:17:11.630ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
9022024-04-03T05:17:11.630ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 7.14\u{b5}s iteration = 12
9032024-04-03T05:17:11.633ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
9042024-04-03T05:17:11.633ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9052024-04-03T05:17:11.633ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
9062024-04-03T05:17:11.633ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9072024-04-03T05:17:11.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9082024-04-03T05:17:11.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9092024-04-03T05:17:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9102024-04-03T05:17:11.634ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
9112024-04-03T05:17:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9122024-04-03T05:17:11.634ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
9132024-04-03T05:17:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9142024-04-03T05:17:11.634ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 12 reason = Timeout
9152024-04-03T05:17:11.634ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9162024-04-03T05:17:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
9172024-04-03T05:17:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9182024-04-03T05:17:11.635ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9192024-04-03T05:17:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9202024-04-03T05:17:11.635ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
9212024-04-03T05:17:11.635ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 1c2282ea-206d-4df3-bfca-a35a264cd464 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
9222024-04-03T05:17:11.635ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
9232024-04-03T05:17:11.635ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9242024-04-03T05:17:11.635ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
9252024-04-03T05:17:11.635ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 5844cfd7-028e-4204-ba41-2d4da75c3d1c dns_zone_name = oxide.test found_so_far = 0 version = 2
9262024-04-03T05:17:11.635ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
9272024-04-03T05:17:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9282024-04-03T05:17:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
9292024-04-03T05:17:11.636ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9302024-04-03T05:17:11.636ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9312024-04-03T05:17:11.637ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9322024-04-03T05:17:11.637ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
9332024-04-03T05:17:11.637ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.698818ms iteration = 32
9342024-04-03T05:17:11.639ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9352024-04-03T05:17:11.639ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9362024-04-03T05:17:11.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9372024-04-03T05:17:11.639ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9382024-04-03T05:17:11.639ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9392024-04-03T05:17:11.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9402024-04-03T05:17:11.640ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9412024-04-03T05:17:11.640ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
9422024-04-03T05:17:11.640ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9432024-04-03T05:17:11.640ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
9442024-04-03T05:17:11.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9452024-04-03T05:17:11.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
9462024-04-03T05:17:11.642ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3424, tv_nsec: 324224801 } })
9472024-04-03T05:17:11.642ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:974 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
9482024-04-03T05:17:11.642ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 5844cfd7-028e-4204-ba41-2d4da75c3d1c dns_zone_name = oxide.test found_so_far = 1 version = 2
9492024-04-03T05:17:11.642ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
9502024-04-03T05:17:11.642ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-04-03 05:12:15.995755 UTC dns_group = external generation = 2
9512024-04-03T05:17:11.642ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 23.860148ms iteration = 7
9522024-04-03T05:17:11.642ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-04-03T05:12:11.760059Z, time_modified: 2024-04-03T05:12:11.760059Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: d254228d-6b17-46e4-bc06-e18669dfc21c, db_silo: Silo { identity: SiloIdentity { id: d254228d-6b17-46e4-bc06-e18669dfc21c, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-04-03T05:12:15.962604Z, time_modified: 2024-04-03T05:12:15.962604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "096b197d9f10cd1b196c898f82b2d3451036c6b5bc8fff5b9af57c633900dba5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: d254228d-6b17-46e4-bc06-e18669dfc21c, db_silo: Silo { identity: SiloIdentity { id: d254228d-6b17-46e4-bc06-e18669dfc21c, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-04-03T05:12:15.962604Z, time_modified: 2024-04-03T05:12:15.962604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "096b197d9f10cd1b196c898f82b2d3451036c6b5bc8fff5b9af57c633900dba5" }] }) }
9532024-04-03T05:17:11.642ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 24.02703ms iteration = 7
9542024-04-03T05:17:11.642ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 1c2282ea-206d-4df3-bfca-a35a264cd464 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
9552024-04-03T05:17:11.642ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
9562024-04-03T05:17:11.642ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-04-03 05:12:15.091055 UTC dns_group = internal generation = 1
9572024-04-03T05:17:11.642ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 24.073851ms iteration = 7
9582024-04-03T05:17:11.643ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 7 reason = Timeout
9592024-04-03T05:17:11.644ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("9f24d5b2-898d-40ee-a459-c6576fb9d418.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c33ab12a-21de-454f-9b76-9a78bfc5c720.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c33ab12a-21de-454f-9b76-9a78bfc5c720.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3424, tv_nsec: 326981458 } })
9602024-04-03T05:17:11.645ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
9612024-04-03T05:17:11.645ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.839555ms iteration = 7
9622024-04-03T05:17:11.671ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 12 reason = Timeout
9632024-04-03T05:17:11.671ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
9642024-04-03T05:17:11.671ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 12 reason = Timeout
9652024-04-03T05:17:11.671ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9662024-04-03T05:17:11.671ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 12 reason = Timeout
9672024-04-03T05:17:11.671ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
9682024-04-03T05:17:11.671ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 12 reason = Timeout
9692024-04-03T05:17:11.671ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
9702024-04-03T05:17:11.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
9712024-04-03T05:17:11.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
9722024-04-03T05:17:11.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9732024-04-03T05:17:11.672ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:982 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 zone_address = fd00:1122:3344:101::2
9742024-04-03T05:17:11.672ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
9752024-04-03T05:17:11.673ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "cac9eb1c-ff6d-4a82-8864-4202f6225306", "content-length": "26", "date": "Wed, 03 Apr 2024 05:17:11 GMT"} })
9762024-04-03T05:17:11.673ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:987 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
9772024-04-03T05:17:11.673ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9782024-04-03T05:17:11.673ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
9792024-04-03T05:17:11.673ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.159509ms iteration = 12
9802024-04-03T05:17:11.674ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3424, tv_nsec: 356292569 } })
9812024-04-03T05:17:11.674ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3424, tv_nsec: 356414930 } })
9822024-04-03T05:17:11.674ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
9832024-04-03T05:17:11.674ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:974
9842024-04-03T05:17:11.674ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:974
9852024-04-03T05:17:11.676ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9862024-04-03T05:17:11.676ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9872024-04-03T05:17:11.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9882024-04-03T05:17:11.677ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9892024-04-03T05:17:11.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9902024-04-03T05:17:11.691ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9912024-04-03T05:17:11.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9922024-04-03T05:17:11.692ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9932024-04-03T05:17:11.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9942024-04-03T05:17:11.693ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9952024-04-03T05:17:11.698ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 54b5d8ce-2f89-4588-80f9-2a71dfa53de1, lookup_type: ById(54b5d8ce-2f89-4588-80f9-2a71dfa53de1) } result = Ok(())
9962024-04-03T05:17:11.698ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9972024-04-03T05:17:11.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9982024-04-03T05:17:11.700ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 28.769033ms iteration = 12
9992024-04-03T05:17:11.777ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
10002024-04-03T05:17:11.777ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
10012024-04-03T05:17:11.777ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "ee0b98c5-621f-4e54-bf16-0bbf18e30fe1", "content-length": "26", "date": "Wed, 03 Apr 2024 05:17:11 GMT"} })
10022024-04-03T05:17:11.777ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
10032024-04-03T05:17:11.779ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
10042024-04-03T05:17:11.876ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
10052024-04-03T05:17:11.876ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
10062024-04-03T05:17:11.877ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "30d655a1-6e98-478c-87ab-4c26dd0f1c25", "content-length": "26", "date": "Wed, 03 Apr 2024 05:17:11 GMT"} })
10072024-04-03T05:17:11.877ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
10082024-04-03T05:17:11.877ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 205.616109ms iteration = 12
10092024-04-03T05:17:11.878ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
10102024-04-03T05:17:11.948ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 9 reason = Timeout
10112024-04-03T05:17:12.020ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
10122024-04-03T05:17:12.021ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "630e5019-ebe7-4c01-9b7b-18760dd1c0ac", "content-length": "1", "date": "Wed, 03 Apr 2024 05:17:11 GMT"} })
10132024-04-03T05:17:12.022ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
10142024-04-03T05:17:12.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
10152024-04-03T05:17:12.024ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 352.376064ms iteration = 12
10162024-04-03T05:17:12.036ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
10172024-04-03T05:17:12.092ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
10182024-04-03T05:17:12.092ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10192024-04-03T05:17:12.092ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "b527faf0-5aa8-4b4f-a3d7-e7b975a4f5c5", "date": "Wed, 03 Apr 2024 05:17:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
10202024-04-03T05:17:12.092ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
10212024-04-03T05:17:12.093ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10222024-04-03T05:17:12.094ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "e1a67c8a-a2a9-45e7-9622-96066952b763", "date": "Wed, 03 Apr 2024 05:17:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
10232024-04-03T05:17:12.094ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
10242024-04-03T05:17:12.094ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:141 port = SwitchPort { id: 9abb0c53-3a74-441c-8a11-a2b0939760e0, rack_id: 38b3f10b-8962-43bd-8d32-8172eeecefb9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(0a16a58f-d519-4d96-87f3-b4294bb539f8) } rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
10252024-04-03T05:17:12.094ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10262024-04-03T05:17:12.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10272024-04-03T05:17:12.096ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
10282024-04-03T05:17:12.096ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 147.548295ms iteration = 9
10292024-04-03T05:17:12.103ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
10302024-04-03T05:17:12.103ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "17f12bcc-3aa4-4f9a-a955-29518dfd180b", "content-length": "121", "date": "Wed, 03 Apr 2024 05:17:11 GMT"} })
10312024-04-03T05:17:12.103ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1418 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
10322024-04-03T05:17:12.103ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1445 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 0a16a58f-d519-4d96-87f3-b4294bb539f8
10332024-04-03T05:17:12.103ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:364 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
10342024-04-03T05:17:12.103ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:368 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
10352024-04-03T05:17:12.103ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:377 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
10362024-04-03T05:17:12.103ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:383 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
10372024-04-03T05:17:12.103ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:403 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
10382024-04-03T05:17:12.103ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10392024-04-03T05:17:12.104ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10402024-04-03T05:17:12.104ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
10412024-04-03T05:17:12.105ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "c4eb1ad9-c3db-4ed2-9a05-b732dfe2b8a1", "content-length": "2", "date": "Wed, 03 Apr 2024 05:17:11 GMT"} })
10422024-04-03T05:17:12.105ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
10432024-04-03T05:17:12.105ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "bc3c7a1b-6300-4e9a-9e61-3e0dccd76b59", "content-length": "52", "date": "Wed, 03 Apr 2024 05:17:11 GMT"} })
10442024-04-03T05:17:12.105ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:452 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
10452024-04-03T05:17:12.105ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
10462024-04-03T05:17:12.155ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "3c28f809-9115-4254-9b46-f4c937f005df", "date": "Wed, 03 Apr 2024 05:17:12 GMT"} })
10472024-04-03T05:17:12.155ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:617 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch = Switch0
10482024-04-03T05:17:12.155ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
10492024-04-03T05:17:12.156ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "4cf7ef4e-951a-432d-8267-1a622a21bcad", "content-length": "452", "date": "Wed, 03 Apr 2024 05:17:12 GMT"} })
10502024-04-03T05:17:12.156ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10512024-04-03T05:17:12.156ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10522024-04-03T05:17:12.159ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10532024-04-03T05:17:12.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10542024-04-03T05:17:12.161ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10552024-04-03T05:17:12.162ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10562024-04-03T05:17:12.163ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10572024-04-03T05:17:12.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10582024-04-03T05:17:12.169ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 534.701053ms iteration = 12
10592024-04-03T05:17:12.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 9 reason = Timeout
10602024-04-03T05:17:12.322ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
10612024-04-03T05:17:12.352ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
10622024-04-03T05:17:12.381ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
10632024-04-03T05:17:12.382ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 237 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 06caa0f0-2130-4352-930d-5d79042bbe99 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
10642024-04-03T05:17:12.382ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "720a17cf-68e5-4f9f-8e65-be9054aedb20", "date": "Wed, 03 Apr 2024 05:17:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
10652024-04-03T05:17:12.382ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
10662024-04-03T05:17:12.383ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "cb5f48f0-a8f1-4648-8067-10a519b0d061", "date": "Wed, 03 Apr 2024 05:17:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
10672024-04-03T05:17:12.383ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
10682024-04-03T05:17:12.384ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "0aca0627-3f28-4623-a0d3-262517e261a0", "date": "Wed, 03 Apr 2024 05:17:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
10692024-04-03T05:17:12.385ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
10702024-04-03T05:17:12.386ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
10712024-04-03T05:17:12.387ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 94.690391ms iteration = 9
10722024-04-03T05:17:21.619ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 33 reason = Timeout
10732024-04-03T05:17:21.619ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10742024-04-03T05:17:21.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10752024-04-03T05:17:21.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10762024-04-03T05:17:21.624ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10772024-04-03T05:17:21.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10782024-04-03T05:17:21.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10792024-04-03T05:17:21.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10802024-04-03T05:17:21.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10812024-04-03T05:17:21.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10822024-04-03T05:17:21.630ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
10832024-04-03T05:17:21.630ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10842024-04-03T05:17:21.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10852024-04-03T05:17:21.634ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
10862024-04-03T05:17:21.634ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.058071ms iteration = 33
10872024-04-03T05:17:22.368ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 146 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = a2dcd1c3-c836-4278-9365-b6ce8c1bd783 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
10882024-04-03T05:17:31.618ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 34 reason = Timeout
10892024-04-03T05:17:31.618ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10902024-04-03T05:17:31.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10912024-04-03T05:17:31.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10922024-04-03T05:17:31.623ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10932024-04-03T05:17:31.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10942024-04-03T05:17:31.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10952024-04-03T05:17:31.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10962024-04-03T05:17:31.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10972024-04-03T05:17:31.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10982024-04-03T05:17:31.629ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
10992024-04-03T05:17:31.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11002024-04-03T05:17:31.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11012024-04-03T05:17:31.634ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
11022024-04-03T05:17:31.634ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.520957ms iteration = 34
11032024-04-03T05:17:32.367ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 125 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = f822a5f5-2040-4215-9fae-eb9f096fed8f response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
11042024-04-03T05:17:41.619ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 35 reason = Timeout
11052024-04-03T05:17:41.619ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11062024-04-03T05:17:41.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11072024-04-03T05:17:41.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11082024-04-03T05:17:41.624ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
11092024-04-03T05:17:41.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11102024-04-03T05:17:41.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11112024-04-03T05:17:41.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11122024-04-03T05:17:41.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11132024-04-03T05:17:41.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11142024-04-03T05:17:41.629ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
11152024-04-03T05:17:41.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11162024-04-03T05:17:41.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11172024-04-03T05:17:41.630ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 13 reason = Timeout
11182024-04-03T05:17:41.630ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
11192024-04-03T05:17:41.630ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
11202024-04-03T05:17:41.630ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 9.2\u{b5}s iteration = 13
11212024-04-03T05:17:41.633ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
11222024-04-03T05:17:41.633ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 14.621375ms iteration = 35
11232024-04-03T05:17:41.634ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 13 reason = Timeout
11242024-04-03T05:17:41.634ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11252024-04-03T05:17:41.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11262024-04-03T05:17:41.636ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11272024-04-03T05:17:41.637ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
11282024-04-03T05:17:41.637ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11292024-04-03T05:17:41.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11302024-04-03T05:17:41.640ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3454, tv_nsec: 321904490 } })
11312024-04-03T05:17:41.641ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:974 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
11322024-04-03T05:17:41.670ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:982 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 zone_address = fd00:1122:3344:101::2
11332024-04-03T05:17:41.670ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11342024-04-03T05:17:41.670ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "a9c8b526-664e-4a6c-981d-1007707db318", "content-length": "26", "date": "Wed, 03 Apr 2024 05:17:41 GMT"} })
11352024-04-03T05:17:41.670ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:987 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11362024-04-03T05:17:41.671ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
11372024-04-03T05:17:41.672ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 13 reason = Timeout
11382024-04-03T05:17:41.672ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 13 reason = Timeout
11392024-04-03T05:17:41.672ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11402024-04-03T05:17:41.672ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
11412024-04-03T05:17:41.672ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 13 reason = Timeout
11422024-04-03T05:17:41.672ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
11432024-04-03T05:17:41.672ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 13 reason = Timeout
11442024-04-03T05:17:41.672ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
11452024-04-03T05:17:41.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11462024-04-03T05:17:41.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11472024-04-03T05:17:41.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
11482024-04-03T05:17:41.674ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11492024-04-03T05:17:41.674ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3454, tv_nsec: 356149126 } })
11502024-04-03T05:17:41.675ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:974
11512024-04-03T05:17:41.675ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
11522024-04-03T05:17:41.675ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.301834ms iteration = 13
11532024-04-03T05:17:41.675ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3454, tv_nsec: 357457413 } })
11542024-04-03T05:17:41.676ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:974
11552024-04-03T05:17:41.677ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
11562024-04-03T05:17:41.677ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11572024-04-03T05:17:41.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11582024-04-03T05:17:41.678ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11592024-04-03T05:17:41.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11602024-04-03T05:17:41.691ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11612024-04-03T05:17:41.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11622024-04-03T05:17:41.692ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11632024-04-03T05:17:41.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11642024-04-03T05:17:41.694ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11652024-04-03T05:17:41.697ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 54b5d8ce-2f89-4588-80f9-2a71dfa53de1, lookup_type: ById(54b5d8ce-2f89-4588-80f9-2a71dfa53de1) } result = Ok(())
11662024-04-03T05:17:41.697ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11672024-04-03T05:17:41.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11682024-04-03T05:17:41.698ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 26.24999ms iteration = 13
11692024-04-03T05:17:41.768ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
11702024-04-03T05:17:41.768ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11712024-04-03T05:17:41.768ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
11722024-04-03T05:17:41.768ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11732024-04-03T05:17:41.768ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "7d169f65-500e-4eb3-abce-7be2011f41da", "content-length": "26", "date": "Wed, 03 Apr 2024 05:17:41 GMT"} })
11742024-04-03T05:17:41.768ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11752024-04-03T05:17:41.768ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "62ed2968-d88e-4e41-bd7f-122ee419b092", "content-length": "26", "date": "Wed, 03 Apr 2024 05:17:41 GMT"} })
11762024-04-03T05:17:41.768ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11772024-04-03T05:17:41.770ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11782024-04-03T05:17:41.770ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11792024-04-03T05:17:41.940ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11802024-04-03T05:17:41.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11812024-04-03T05:17:41.943ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:141 port = SwitchPort { id: 9abb0c53-3a74-441c-8a11-a2b0939760e0, rack_id: 38b3f10b-8962-43bd-8d32-8172eeecefb9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(0a16a58f-d519-4d96-87f3-b4294bb539f8) } rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
11822024-04-03T05:17:41.943ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11832024-04-03T05:17:41.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11842024-04-03T05:17:41.948ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
11852024-04-03T05:17:41.949ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "881de989-f1db-47b0-a005-adc87b7b109b", "content-length": "121", "date": "Wed, 03 Apr 2024 05:17:41 GMT"} })
11862024-04-03T05:17:41.949ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1418 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
11872024-04-03T05:17:41.949ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1445 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 0a16a58f-d519-4d96-87f3-b4294bb539f8
11882024-04-03T05:17:41.950ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:364 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
11892024-04-03T05:17:41.950ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:368 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
11902024-04-03T05:17:41.950ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:377 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
11912024-04-03T05:17:41.950ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:383 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
11922024-04-03T05:17:41.950ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:403 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
11932024-04-03T05:17:41.950ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11942024-04-03T05:17:41.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11952024-04-03T05:17:41.951ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
11962024-04-03T05:17:41.951ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "031b169c-10e7-44db-a813-e72900c129e0", "content-length": "2", "date": "Wed, 03 Apr 2024 05:17:41 GMT"} })
11972024-04-03T05:17:41.951ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
11982024-04-03T05:17:41.951ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2a8ccec2-8102-415e-8d6b-d1473dcdbb34", "content-length": "52", "date": "Wed, 03 Apr 2024 05:17:41 GMT"} })
11992024-04-03T05:17:41.951ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:452 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
12002024-04-03T05:17:41.951ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
12012024-04-03T05:17:41.956ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 283.920892ms iteration = 13
12022024-04-03T05:17:41.958ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
12032024-04-03T05:17:41.959ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "a1539235-3b14-47b1-8008-8f7d25f1f569", "content-length": "1", "date": "Wed, 03 Apr 2024 05:17:41 GMT"} })
12042024-04-03T05:17:41.960ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
12052024-04-03T05:17:41.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
12062024-04-03T05:17:41.962ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 290.079194ms iteration = 13
12072024-04-03T05:17:42.029ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "c510a5d4-ebbe-4bcb-982d-22a2082a284a", "date": "Wed, 03 Apr 2024 05:17:41 GMT"} })
12082024-04-03T05:17:42.029ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:617 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch = Switch0
12092024-04-03T05:17:42.029ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
12102024-04-03T05:17:42.032ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "c8061e7e-b29e-4458-ae7c-b37527b58567", "content-length": "452", "date": "Wed, 03 Apr 2024 05:17:41 GMT"} })
12112024-04-03T05:17:42.032ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12122024-04-03T05:17:42.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12132024-04-03T05:17:42.035ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12142024-04-03T05:17:42.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12152024-04-03T05:17:42.038ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12162024-04-03T05:17:42.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12172024-04-03T05:17:42.039ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12182024-04-03T05:17:42.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12192024-04-03T05:17:42.046ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 411.835856ms iteration = 13
12202024-04-03T05:17:42.367ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 142 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 4216ca2d-83f0-4ecd-9e85-e324242f7d93 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
12212024-04-03T05:17:51.619ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 36 reason = Timeout
12222024-04-03T05:17:51.619ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12232024-04-03T05:17:51.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12242024-04-03T05:17:51.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12252024-04-03T05:17:51.623ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12262024-04-03T05:17:51.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12272024-04-03T05:17:51.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12282024-04-03T05:17:51.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12292024-04-03T05:17:51.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12302024-04-03T05:17:51.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12312024-04-03T05:17:51.629ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
12322024-04-03T05:17:51.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12332024-04-03T05:17:51.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12342024-04-03T05:17:51.635ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
12352024-04-03T05:17:51.635ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.246777ms iteration = 36
12362024-04-03T05:17:52.368ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 121 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = a9addcc8-e09a-45ad-b0bf-9c483990fe65 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
12372024-04-03T05:18:01.619ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 37 reason = Timeout
12382024-04-03T05:18:01.619ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12392024-04-03T05:18:01.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12402024-04-03T05:18:01.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12412024-04-03T05:18:01.624ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12422024-04-03T05:18:01.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12432024-04-03T05:18:01.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12442024-04-03T05:18:01.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12452024-04-03T05:18:01.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12462024-04-03T05:18:01.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12472024-04-03T05:18:01.629ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
12482024-04-03T05:18:01.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12492024-04-03T05:18:01.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12502024-04-03T05:18:01.634ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
12512024-04-03T05:18:01.634ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.072431ms iteration = 37
12522024-04-03T05:18:02.368ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 141 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 1b32a4f9-d8d4-478c-a996-224d558ed9b7 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
12532024-04-03T05:18:11.619ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 38 reason = Timeout
12542024-04-03T05:18:11.619ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12552024-04-03T05:18:11.619ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
12562024-04-03T05:18:11.619ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
12572024-04-03T05:18:11.619ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
12582024-04-03T05:18:11.619ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
12592024-04-03T05:18:11.619ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
12602024-04-03T05:18:11.619ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
12612024-04-03T05:18:11.619ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
12622024-04-03T05:18:11.619ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
12632024-04-03T05:18:11.619ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 8 reason = Timeout
12642024-04-03T05:18:11.619ZWARNnexus (BackgroundTasks): Metric producer GC: statically disabled pending omicron#5284 background_task = metrics_producer_gc file = nexus/src/app/background/metrics_producer_gc.rs:43
12652024-04-03T05:18:11.619ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 18.08\u{b5}s iteration = 8
12662024-04-03T05:18:11.619ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
12672024-04-03T05:18:11.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12682024-04-03T05:18:11.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
12692024-04-03T05:18:11.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
12702024-04-03T05:18:11.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
12712024-04-03T05:18:11.620ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("174b6905-67c4-445e-8ee0-92f9d656bfce.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1b4f3db-685d-4e05-94cb-90ed84711941.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a2b50f43-9fdf-42d4-88bc-53143a838677.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("a2b50f43-9fdf-42d4-88bc-53143a838677.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 3484, tv_nsec: 301632458 } })
12722024-04-03T05:18:11.620ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
12732024-04-03T05:18:11.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
12742024-04-03T05:18:11.621ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
12752024-04-03T05:18:11.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
12762024-04-03T05:18:11.621ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
12772024-04-03T05:18:11.621ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.49323ms iteration = 8
12782024-04-03T05:18:11.622ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12792024-04-03T05:18:11.622ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12802024-04-03T05:18:11.622ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12812024-04-03T05:18:11.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12822024-04-03T05:18:11.625ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
12832024-04-03T05:18:11.625ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
12842024-04-03T05:18:11.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
12852024-04-03T05:18:11.627ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
12862024-04-03T05:18:11.627ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
12872024-04-03T05:18:11.627ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
12882024-04-03T05:18:11.627ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12892024-04-03T05:18:11.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12902024-04-03T05:18:11.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
12912024-04-03T05:18:11.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12922024-04-03T05:18:11.628ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
12932024-04-03T05:18:11.628ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
12942024-04-03T05:18:11.628ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
12952024-04-03T05:18:11.628ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
12962024-04-03T05:18:11.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
12972024-04-03T05:18:11.628ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
12982024-04-03T05:18:11.628ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
12992024-04-03T05:18:11.628ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
13002024-04-03T05:18:11.628ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13012024-04-03T05:18:11.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13022024-04-03T05:18:11.628ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13032024-04-03T05:18:11.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13042024-04-03T05:18:11.629ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13052024-04-03T05:18:11.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13062024-04-03T05:18:11.629ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13072024-04-03T05:18:11.629ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
13082024-04-03T05:18:11.629ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:65 reason = blueprint disabled target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e
13092024-04-03T05:18:11.629ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.39\u{b5}s iteration = 9
13102024-04-03T05:18:11.630ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13112024-04-03T05:18:11.631ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
13122024-04-03T05:18:11.631ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
13132024-04-03T05:18:11.631ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
13142024-04-03T05:18:11.631ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 8.15\u{b5}s iteration = 14
13152024-04-03T05:18:11.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
13162024-04-03T05:18:11.632ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13172024-04-03T05:18:11.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13182024-04-03T05:18:11.633ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
13192024-04-03T05:18:11.633ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13202024-04-03T05:18:11.633ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
13212024-04-03T05:18:11.633ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13222024-04-03T05:18:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13232024-04-03T05:18:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13242024-04-03T05:18:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
13252024-04-03T05:18:11.634ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13262024-04-03T05:18:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13272024-04-03T05:18:11.634ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13282024-04-03T05:18:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13292024-04-03T05:18:11.635ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 14 reason = Timeout
13302024-04-03T05:18:11.635ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13312024-04-03T05:18:11.635ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
13322024-04-03T05:18:11.635ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 1c2282ea-206d-4df3-bfca-a35a264cd464 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
13332024-04-03T05:18:11.635ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13342024-04-03T05:18:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13352024-04-03T05:18:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13362024-04-03T05:18:11.635ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13372024-04-03T05:18:11.635ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
13382024-04-03T05:18:11.635ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 5844cfd7-028e-4204-ba41-2d4da75c3d1c dns_zone_name = oxide.test found_so_far = 0 version = 2
13392024-04-03T05:18:11.635ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13402024-04-03T05:18:11.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13412024-04-03T05:18:11.636ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13422024-04-03T05:18:11.636ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13432024-04-03T05:18:11.637ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13442024-04-03T05:18:11.638ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
13452024-04-03T05:18:11.638ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.671648ms iteration = 38
13462024-04-03T05:18:11.638ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
13472024-04-03T05:18:11.638ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13482024-04-03T05:18:11.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13492024-04-03T05:18:11.639ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
13502024-04-03T05:18:11.639ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13512024-04-03T05:18:11.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13522024-04-03T05:18:11.640ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
13532024-04-03T05:18:11.640ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13542024-04-03T05:18:11.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13552024-04-03T05:18:11.641ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
13562024-04-03T05:18:11.641ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13572024-04-03T05:18:11.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13582024-04-03T05:18:11.641ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-04-03T05:12:11.760059Z, time_modified: 2024-04-03T05:12:11.760059Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: d254228d-6b17-46e4-bc06-e18669dfc21c, db_silo: Silo { identity: SiloIdentity { id: d254228d-6b17-46e4-bc06-e18669dfc21c, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-04-03T05:12:15.962604Z, time_modified: 2024-04-03T05:12:15.962604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "096b197d9f10cd1b196c898f82b2d3451036c6b5bc8fff5b9af57c633900dba5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: d254228d-6b17-46e4-bc06-e18669dfc21c, db_silo: Silo { identity: SiloIdentity { id: d254228d-6b17-46e4-bc06-e18669dfc21c, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-04-03T05:12:15.962604Z, time_modified: 2024-04-03T05:12:15.962604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "096b197d9f10cd1b196c898f82b2d3451036c6b5bc8fff5b9af57c633900dba5" }] }) } file = nexus/src/app/background/external_endpoints.rs:101
13592024-04-03T05:18:11.641ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 21.601947ms iteration = 8
13602024-04-03T05:18:11.641ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3484, tv_nsec: 322588527 } })
13612024-04-03T05:18:11.641ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:974 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
13622024-04-03T05:18:11.642ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 1c2282ea-206d-4df3-bfca-a35a264cd464 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
13632024-04-03T05:18:11.642ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
13642024-04-03T05:18:11.642ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-04-03 05:12:15.091055 UTC dns_group = internal generation = 1
13652024-04-03T05:18:11.642ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 22.737993ms iteration = 8
13662024-04-03T05:18:11.642ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 5844cfd7-028e-4204-ba41-2d4da75c3d1c dns_zone_name = oxide.test found_so_far = 1 version = 2
13672024-04-03T05:18:11.642ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
13682024-04-03T05:18:11.642ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-04-03 05:12:15.995755 UTC dns_group = external generation = 2
13692024-04-03T05:18:11.642ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 22.998946ms iteration = 8
13702024-04-03T05:18:11.644ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
13712024-04-03T05:18:11.645ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("9f24d5b2-898d-40ee-a459-c6576fb9d418.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c33ab12a-21de-454f-9b76-9a78bfc5c720.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c33ab12a-21de-454f-9b76-9a78bfc5c720.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3484, tv_nsec: 326219516 } })
13722024-04-03T05:18:11.646ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
13732024-04-03T05:18:11.646ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.337818ms iteration = 8
13742024-04-03T05:18:11.671ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:982 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 zone_address = fd00:1122:3344:101::2
13752024-04-03T05:18:11.671ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
13762024-04-03T05:18:11.671ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "6f72e374-3d99-465c-bc67-9506bdd440f7", "content-length": "26", "date": "Wed, 03 Apr 2024 05:18:11 GMT"} })
13772024-04-03T05:18:11.671ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:987 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
13782024-04-03T05:18:11.673ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
13792024-04-03T05:18:11.673ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
13802024-04-03T05:18:11.673ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
13812024-04-03T05:18:11.673ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
13822024-04-03T05:18:11.673ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
13832024-04-03T05:18:11.673ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
13842024-04-03T05:18:11.673ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
13852024-04-03T05:18:11.673ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13862024-04-03T05:18:11.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
13872024-04-03T05:18:11.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
13882024-04-03T05:18:11.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13892024-04-03T05:18:11.674ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
13902024-04-03T05:18:11.674ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13912024-04-03T05:18:11.675ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3484, tv_nsec: 356434098 } })
13922024-04-03T05:18:11.675ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
13932024-04-03T05:18:11.675ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.385942ms iteration = 14
13942024-04-03T05:18:11.675ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3484, tv_nsec: 356558150 } })
13952024-04-03T05:18:11.675ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:974
13962024-04-03T05:18:11.676ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:974
13972024-04-03T05:18:11.677ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
13982024-04-03T05:18:11.677ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13992024-04-03T05:18:11.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14002024-04-03T05:18:11.678ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14012024-04-03T05:18:11.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14022024-04-03T05:18:11.692ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14032024-04-03T05:18:11.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14042024-04-03T05:18:11.693ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14052024-04-03T05:18:11.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14062024-04-03T05:18:11.694ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14072024-04-03T05:18:11.698ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 54b5d8ce-2f89-4588-80f9-2a71dfa53de1, lookup_type: ById(54b5d8ce-2f89-4588-80f9-2a71dfa53de1) } result = Ok(())
14082024-04-03T05:18:11.698ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14092024-04-03T05:18:11.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14102024-04-03T05:18:11.699ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 26.421442ms iteration = 14
14112024-04-03T05:18:11.763ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
14122024-04-03T05:18:11.763ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14132024-04-03T05:18:11.763ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "607dc347-0113-42dd-8123-b0b4b2b82dbc", "content-length": "26", "date": "Wed, 03 Apr 2024 05:18:11 GMT"} })
14142024-04-03T05:18:11.763ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14152024-04-03T05:18:11.766ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
14162024-04-03T05:18:11.862ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
14172024-04-03T05:18:11.863ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "fc834ef1-919c-48f1-bd11-0e6f9acca1fb", "content-length": "1", "date": "Wed, 03 Apr 2024 05:18:11 GMT"} })
14182024-04-03T05:18:11.865ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
14192024-04-03T05:18:11.866ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
14202024-04-03T05:18:11.867ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 194.720944ms iteration = 14
14212024-04-03T05:18:11.867ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
14222024-04-03T05:18:11.867ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14232024-04-03T05:18:11.868ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e8b3cf15-5217-48b3-9d29-c3ed879ff1eb", "content-length": "26", "date": "Wed, 03 Apr 2024 05:18:11 GMT"} })
14242024-04-03T05:18:11.868ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14252024-04-03T05:18:11.869ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
14262024-04-03T05:18:11.930ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 257.778924ms iteration = 14
14272024-04-03T05:18:11.949ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
14282024-04-03T05:18:11.954ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14292024-04-03T05:18:11.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14302024-04-03T05:18:11.957ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:141 port = SwitchPort { id: 9abb0c53-3a74-441c-8a11-a2b0939760e0, rack_id: 38b3f10b-8962-43bd-8d32-8172eeecefb9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(0a16a58f-d519-4d96-87f3-b4294bb539f8) } rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
14312024-04-03T05:18:11.957ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14322024-04-03T05:18:11.957ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14332024-04-03T05:18:11.963ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
14342024-04-03T05:18:11.963ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "4914d381-e8e6-474f-8d16-3a97431cc56d", "content-length": "121", "date": "Wed, 03 Apr 2024 05:18:11 GMT"} })
14352024-04-03T05:18:11.963ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1418 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
14362024-04-03T05:18:11.963ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1445 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 0a16a58f-d519-4d96-87f3-b4294bb539f8
14372024-04-03T05:18:11.964ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:364 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
14382024-04-03T05:18:11.964ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:368 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
14392024-04-03T05:18:11.964ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:377 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
14402024-04-03T05:18:11.964ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:383 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
14412024-04-03T05:18:11.964ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:403 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
14422024-04-03T05:18:11.964ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14432024-04-03T05:18:11.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14442024-04-03T05:18:11.965ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
14452024-04-03T05:18:11.965ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "071aa91c-8cc7-49f8-9682-65d65718baa3", "content-length": "2", "date": "Wed, 03 Apr 2024 05:18:11 GMT"} })
14462024-04-03T05:18:11.965ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
14472024-04-03T05:18:11.966ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9a72e55e-65c0-4cf3-969f-1bdaadcd27ce", "content-length": "52", "date": "Wed, 03 Apr 2024 05:18:11 GMT"} })
14482024-04-03T05:18:11.966ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:452 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
14492024-04-03T05:18:11.966ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
14502024-04-03T05:18:11.980ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
14512024-04-03T05:18:12.009ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
14522024-04-03T05:18:12.011ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "7810f40d-9ce9-46fd-a653-dff8e62b7c50", "date": "Wed, 03 Apr 2024 05:18:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14532024-04-03T05:18:12.011ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14542024-04-03T05:18:12.012ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "9df99496-4380-4ecf-92e7-d2f4589bb00f", "date": "Wed, 03 Apr 2024 05:18:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14552024-04-03T05:18:12.012ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14562024-04-03T05:18:12.013ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14572024-04-03T05:18:12.013ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 63.647438ms iteration = 10
14582024-04-03T05:18:12.018ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "b4d68a24-bf9e-4bc4-9242-8c04626f7a6c", "date": "Wed, 03 Apr 2024 05:18:11 GMT"} })
14592024-04-03T05:18:12.018ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:617 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch = Switch0
14602024-04-03T05:18:12.018ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
14612024-04-03T05:18:12.018ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "08e7c2c0-2c31-46b1-a67c-9d7f893961fe", "content-length": "452", "date": "Wed, 03 Apr 2024 05:18:11 GMT"} })
14622024-04-03T05:18:12.018ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14632024-04-03T05:18:12.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14642024-04-03T05:18:12.021ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14652024-04-03T05:18:12.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14662024-04-03T05:18:12.024ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14672024-04-03T05:18:12.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14682024-04-03T05:18:12.026ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14692024-04-03T05:18:12.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14702024-04-03T05:18:12.034ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 398.877094ms iteration = 14
14712024-04-03T05:18:12.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
14722024-04-03T05:18:12.323ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
14732024-04-03T05:18:12.353ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
14742024-04-03T05:18:12.383ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
14752024-04-03T05:18:12.383ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 130 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 588a2e45-1a4a-4b84-97de-9c2cb4c4e2fa response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
14762024-04-03T05:18:12.383ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "f9279d4c-636c-4128-a9e2-21a24293d304", "date": "Wed, 03 Apr 2024 05:18:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
14772024-04-03T05:18:12.383ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
14782024-04-03T05:18:12.385ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "3ed0da32-0a83-413d-96fb-eb61e724a419", "date": "Wed, 03 Apr 2024 05:18:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
14792024-04-03T05:18:12.385ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
14802024-04-03T05:18:12.386ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "07ba18c9-4b3f-41f6-a138-6e22e558b329", "date": "Wed, 03 Apr 2024 05:18:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
14812024-04-03T05:18:12.386ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
14822024-04-03T05:18:12.388ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
14832024-04-03T05:18:12.388ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 95.582064ms iteration = 10
14842024-04-03T05:18:21.620ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 39 reason = Timeout
14852024-04-03T05:18:21.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14862024-04-03T05:18:21.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14872024-04-03T05:18:21.622ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14882024-04-03T05:18:21.625ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
14892024-04-03T05:18:21.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14902024-04-03T05:18:21.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14912024-04-03T05:18:21.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14922024-04-03T05:18:21.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14932024-04-03T05:18:21.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14942024-04-03T05:18:21.631ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
14952024-04-03T05:18:21.631ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14962024-04-03T05:18:21.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14972024-04-03T05:18:21.636ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
14982024-04-03T05:18:21.636ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.059904ms iteration = 39
14992024-04-03T05:18:22.368ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 122 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 7e239993-8051-4746-b984-40ad9e105d91 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
15002024-04-03T05:18:31.619ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
15012024-04-03T05:18:31.619ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15022024-04-03T05:18:31.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15032024-04-03T05:18:31.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15042024-04-03T05:18:31.624ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15052024-04-03T05:18:31.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15062024-04-03T05:18:31.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15072024-04-03T05:18:31.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15082024-04-03T05:18:31.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15092024-04-03T05:18:31.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15102024-04-03T05:18:31.630ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
15112024-04-03T05:18:31.630ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15122024-04-03T05:18:31.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15132024-04-03T05:18:31.634ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
15142024-04-03T05:18:31.634ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.092181ms iteration = 40
15152024-04-03T05:18:32.368ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 145 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 08bb31db-89f9-4b29-b5e8-5de2b8333d27 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
15162024-04-03T05:18:41.619ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
15172024-04-03T05:18:41.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15182024-04-03T05:18:41.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15192024-04-03T05:18:41.622ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15202024-04-03T05:18:41.625ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15212024-04-03T05:18:41.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15222024-04-03T05:18:41.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15232024-04-03T05:18:41.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15242024-04-03T05:18:41.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15252024-04-03T05:18:41.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15262024-04-03T05:18:41.631ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
15272024-04-03T05:18:41.631ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15282024-04-03T05:18:41.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15292024-04-03T05:18:41.631ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
15302024-04-03T05:18:41.631ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
15312024-04-03T05:18:41.631ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
15322024-04-03T05:18:41.631ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 9.11\u{b5}s iteration = 15
15332024-04-03T05:18:41.635ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
15342024-04-03T05:18:41.635ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.522136ms iteration = 41
15352024-04-03T05:18:41.635ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 15 reason = Timeout
15362024-04-03T05:18:41.635ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15372024-04-03T05:18:41.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15382024-04-03T05:18:41.637ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15392024-04-03T05:18:41.638ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
15402024-04-03T05:18:41.638ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15412024-04-03T05:18:41.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15422024-04-03T05:18:41.641ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3514, tv_nsec: 322199342 } })
15432024-04-03T05:18:41.641ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:974 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
15442024-04-03T05:18:41.671ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:982 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 zone_address = fd00:1122:3344:101::2
15452024-04-03T05:18:41.671ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
15462024-04-03T05:18:41.671ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "acb8159c-9c97-4bd7-9277-5533c5094a62", "content-length": "26", "date": "Wed, 03 Apr 2024 05:18:41 GMT"} })
15472024-04-03T05:18:41.671ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:987 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
15482024-04-03T05:18:41.672ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
15492024-04-03T05:18:41.672ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
15502024-04-03T05:18:41.672ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
15512024-04-03T05:18:41.672ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
15522024-04-03T05:18:41.672ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15532024-04-03T05:18:41.672ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
15542024-04-03T05:18:41.672ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
15552024-04-03T05:18:41.672ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
15562024-04-03T05:18:41.672ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
15572024-04-03T05:18:41.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15582024-04-03T05:18:41.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
15592024-04-03T05:18:41.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
15602024-04-03T05:18:41.674ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15612024-04-03T05:18:41.675ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3514, tv_nsec: 355948642 } })
15622024-04-03T05:18:41.675ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
15632024-04-03T05:18:41.675ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.837978ms iteration = 15
15642024-04-03T05:18:41.676ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:974
15652024-04-03T05:18:41.677ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
15662024-04-03T05:18:41.677ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15672024-04-03T05:18:41.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15682024-04-03T05:18:41.677ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3514, tv_nsec: 358282783 } })
15692024-04-03T05:18:41.678ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:974
15702024-04-03T05:18:41.678ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15712024-04-03T05:18:41.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15722024-04-03T05:18:41.691ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15732024-04-03T05:18:41.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15742024-04-03T05:18:41.692ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15752024-04-03T05:18:41.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15762024-04-03T05:18:41.693ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15772024-04-03T05:18:41.696ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 54b5d8ce-2f89-4588-80f9-2a71dfa53de1, lookup_type: ById(54b5d8ce-2f89-4588-80f9-2a71dfa53de1) } result = Ok(())
15782024-04-03T05:18:41.696ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15792024-04-03T05:18:41.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15802024-04-03T05:18:41.698ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 25.664772ms iteration = 15
15812024-04-03T05:18:41.776ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
15822024-04-03T05:18:41.776ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
15832024-04-03T05:18:41.776ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "ae8845dc-1297-4ad2-beb5-ce081fd0df00", "content-length": "26", "date": "Wed, 03 Apr 2024 05:18:41 GMT"} })
15842024-04-03T05:18:41.776ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
15852024-04-03T05:18:41.777ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
15862024-04-03T05:18:41.850ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
15872024-04-03T05:18:41.850ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
15882024-04-03T05:18:41.850ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "1143072c-9204-452e-ac92-44e2a101bb9e", "content-length": "26", "date": "Wed, 03 Apr 2024 05:18:41 GMT"} })
15892024-04-03T05:18:41.850ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
15902024-04-03T05:18:41.852ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
15912024-04-03T05:18:41.856ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
15922024-04-03T05:18:41.856ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "719769f3-2174-4a8b-9ced-cac9e5849d97", "content-length": "2", "date": "Wed, 03 Apr 2024 05:18:41 GMT"} })
15932024-04-03T05:18:41.857ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
15942024-04-03T05:18:41.858ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
15952024-04-03T05:18:41.859ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 186.793138ms iteration = 15
15962024-04-03T05:18:41.916ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 243.880369ms iteration = 15
15972024-04-03T05:18:41.950ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15982024-04-03T05:18:41.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15992024-04-03T05:18:41.953ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:141 port = SwitchPort { id: 9abb0c53-3a74-441c-8a11-a2b0939760e0, rack_id: 38b3f10b-8962-43bd-8d32-8172eeecefb9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(0a16a58f-d519-4d96-87f3-b4294bb539f8) } rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
16002024-04-03T05:18:41.953ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16012024-04-03T05:18:41.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16022024-04-03T05:18:41.959ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
16032024-04-03T05:18:41.959ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f3c8c124-badf-4b78-821a-0e8c52e1bc3d", "content-length": "121", "date": "Wed, 03 Apr 2024 05:18:41 GMT"} })
16042024-04-03T05:18:41.959ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1418 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
16052024-04-03T05:18:41.959ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1445 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 0a16a58f-d519-4d96-87f3-b4294bb539f8
16062024-04-03T05:18:41.960ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:364 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
16072024-04-03T05:18:41.960ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:368 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
16082024-04-03T05:18:41.960ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:377 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
16092024-04-03T05:18:41.960ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:383 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
16102024-04-03T05:18:41.960ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:403 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
16112024-04-03T05:18:41.960ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16122024-04-03T05:18:41.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16132024-04-03T05:18:41.961ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
16142024-04-03T05:18:41.961ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "10fcde60-a638-4cdb-aa24-f55ee8766d0a", "content-length": "2", "date": "Wed, 03 Apr 2024 05:18:41 GMT"} })
16152024-04-03T05:18:41.961ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
16162024-04-03T05:18:41.961ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "a7521e41-47f2-46df-a39c-ae6fb8e11add", "content-length": "52", "date": "Wed, 03 Apr 2024 05:18:41 GMT"} })
16172024-04-03T05:18:41.961ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:452 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
16182024-04-03T05:18:41.961ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
16192024-04-03T05:18:42.011ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "ca4c3115-d45c-45d2-83d6-071e8c4e2a8e", "date": "Wed, 03 Apr 2024 05:18:41 GMT"} })
16202024-04-03T05:18:42.011ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:617 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch = Switch0
16212024-04-03T05:18:42.011ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
16222024-04-03T05:18:42.015ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e7c1d052-07ef-459b-99ce-a07d9bc81557", "content-length": "452", "date": "Wed, 03 Apr 2024 05:18:42 GMT"} })
16232024-04-03T05:18:42.015ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16242024-04-03T05:18:42.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16252024-04-03T05:18:42.019ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16262024-04-03T05:18:42.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16272024-04-03T05:18:42.021ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16282024-04-03T05:18:42.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16292024-04-03T05:18:42.023ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16302024-04-03T05:18:42.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16312024-04-03T05:18:42.030ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 394.906181ms iteration = 15
16322024-04-03T05:18:42.370ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 914 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 4d884c7c-93fc-43ea-a76f-ef5f50905514 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
16332024-04-03T05:18:51.620ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
16342024-04-03T05:18:51.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16352024-04-03T05:18:51.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16362024-04-03T05:18:51.622ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16372024-04-03T05:18:51.625ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
16382024-04-03T05:18:51.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16392024-04-03T05:18:51.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16402024-04-03T05:18:51.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16412024-04-03T05:18:51.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16422024-04-03T05:18:51.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16432024-04-03T05:18:51.631ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
16442024-04-03T05:18:51.631ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16452024-04-03T05:18:51.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16462024-04-03T05:18:51.636ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
16472024-04-03T05:18:51.636ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.889361ms iteration = 42
16482024-04-03T05:18:52.369ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 148 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 3272e6d5-5443-4539-835e-cb4253d547a3 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
16492024-04-03T05:19:01.620ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
16502024-04-03T05:19:01.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16512024-04-03T05:19:01.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16522024-04-03T05:19:01.622ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16532024-04-03T05:19:01.624ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
16542024-04-03T05:19:01.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16552024-04-03T05:19:01.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16562024-04-03T05:19:01.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16572024-04-03T05:19:01.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16582024-04-03T05:19:01.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16592024-04-03T05:19:01.630ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
16602024-04-03T05:19:01.630ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16612024-04-03T05:19:01.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16622024-04-03T05:19:01.635ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
16632024-04-03T05:19:01.635ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.188502ms iteration = 43
16642024-04-03T05:19:02.369ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 130 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 33aad5f5-e643-4159-be23-47f80d7616da response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
16652024-04-03T05:19:11.620ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
16662024-04-03T05:19:11.620ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
16672024-04-03T05:19:11.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16682024-04-03T05:19:11.620ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
16692024-04-03T05:19:11.620ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
16702024-04-03T05:19:11.620ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
16712024-04-03T05:19:11.620ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
16722024-04-03T05:19:11.620ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
16732024-04-03T05:19:11.620ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
16742024-04-03T05:19:11.620ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 9 reason = Timeout
16752024-04-03T05:19:11.620ZWARNnexus (BackgroundTasks): Metric producer GC: statically disabled pending omicron#5284 background_task = metrics_producer_gc file = nexus/src/app/background/metrics_producer_gc.rs:43
16762024-04-03T05:19:11.620ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
16772024-04-03T05:19:11.620ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 9.11\u{b5}s iteration = 9
16782024-04-03T05:19:11.620ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
16792024-04-03T05:19:11.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
16802024-04-03T05:19:11.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
16812024-04-03T05:19:11.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16822024-04-03T05:19:11.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
16832024-04-03T05:19:11.621ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("174b6905-67c4-445e-8ee0-92f9d656bfce.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1b4f3db-685d-4e05-94cb-90ed84711941.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a2b50f43-9fdf-42d4-88bc-53143a838677.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("a2b50f43-9fdf-42d4-88bc-53143a838677.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 3544, tv_nsec: 301744769 } })
16842024-04-03T05:19:11.622ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
16852024-04-03T05:19:11.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
16862024-04-03T05:19:11.622ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
16872024-04-03T05:19:11.622ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
16882024-04-03T05:19:11.622ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 2.029527ms iteration = 9
16892024-04-03T05:19:11.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
16902024-04-03T05:19:11.623ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16912024-04-03T05:19:11.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16922024-04-03T05:19:11.623ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16932024-04-03T05:19:11.624ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16942024-04-03T05:19:11.625ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
16952024-04-03T05:19:11.625ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
16962024-04-03T05:19:11.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
16972024-04-03T05:19:11.627ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
16982024-04-03T05:19:11.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
16992024-04-03T05:19:11.628ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
17002024-04-03T05:19:11.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17012024-04-03T05:19:11.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17022024-04-03T05:19:11.628ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
17032024-04-03T05:19:11.628ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
17042024-04-03T05:19:11.629ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17052024-04-03T05:19:11.629ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
17062024-04-03T05:19:11.629ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
17072024-04-03T05:19:11.629ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
17082024-04-03T05:19:11.629ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17092024-04-03T05:19:11.629ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
17102024-04-03T05:19:11.629ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
17112024-04-03T05:19:11.629ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
17122024-04-03T05:19:11.629ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17132024-04-03T05:19:11.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17142024-04-03T05:19:11.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17152024-04-03T05:19:11.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17162024-04-03T05:19:11.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17172024-04-03T05:19:11.630ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
17182024-04-03T05:19:11.630ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:65 reason = blueprint disabled target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e
17192024-04-03T05:19:11.630ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 16.121\u{b5}s iteration = 10
17202024-04-03T05:19:11.631ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17212024-04-03T05:19:11.631ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17222024-04-03T05:19:11.631ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17232024-04-03T05:19:11.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
17242024-04-03T05:19:11.632ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
17252024-04-03T05:19:11.632ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
17262024-04-03T05:19:11.632ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
17272024-04-03T05:19:11.632ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
17282024-04-03T05:19:11.632ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 4.29\u{b5}s iteration = 16
17292024-04-03T05:19:11.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
17302024-04-03T05:19:11.634ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
17312024-04-03T05:19:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
17322024-04-03T05:19:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
17332024-04-03T05:19:11.635ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17342024-04-03T05:19:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17352024-04-03T05:19:11.635ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17362024-04-03T05:19:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
17372024-04-03T05:19:11.635ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17382024-04-03T05:19:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
17392024-04-03T05:19:11.635ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17402024-04-03T05:19:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17412024-04-03T05:19:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17422024-04-03T05:19:11.636ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 16 reason = Timeout
17432024-04-03T05:19:11.636ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17442024-04-03T05:19:11.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17452024-04-03T05:19:11.637ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
17462024-04-03T05:19:11.637ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 5844cfd7-028e-4204-ba41-2d4da75c3d1c dns_zone_name = oxide.test found_so_far = 0 version = 2
17472024-04-03T05:19:11.637ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17482024-04-03T05:19:11.637ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
17492024-04-03T05:19:11.637ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 1c2282ea-206d-4df3-bfca-a35a264cd464 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
17502024-04-03T05:19:11.637ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17512024-04-03T05:19:11.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17522024-04-03T05:19:11.637ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
17532024-04-03T05:19:11.637ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
17542024-04-03T05:19:11.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17552024-04-03T05:19:11.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
17562024-04-03T05:19:11.637ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17572024-04-03T05:19:11.638ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17582024-04-03T05:19:11.638ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17592024-04-03T05:19:11.639ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
17602024-04-03T05:19:11.639ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.049873ms iteration = 44
17612024-04-03T05:19:11.640ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
17622024-04-03T05:19:11.640ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17632024-04-03T05:19:11.640ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-04-03T05:12:11.760059Z, time_modified: 2024-04-03T05:12:11.760059Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: d254228d-6b17-46e4-bc06-e18669dfc21c, db_silo: Silo { identity: SiloIdentity { id: d254228d-6b17-46e4-bc06-e18669dfc21c, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-04-03T05:12:15.962604Z, time_modified: 2024-04-03T05:12:15.962604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "096b197d9f10cd1b196c898f82b2d3451036c6b5bc8fff5b9af57c633900dba5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: d254228d-6b17-46e4-bc06-e18669dfc21c, db_silo: Silo { identity: SiloIdentity { id: d254228d-6b17-46e4-bc06-e18669dfc21c, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-04-03T05:12:15.962604Z, time_modified: 2024-04-03T05:12:15.962604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "096b197d9f10cd1b196c898f82b2d3451036c6b5bc8fff5b9af57c633900dba5" }] }) }
17642024-04-03T05:19:11.640ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 20.126778ms iteration = 9
17652024-04-03T05:19:11.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17662024-04-03T05:19:11.642ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
17672024-04-03T05:19:11.642ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17682024-04-03T05:19:11.642ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17692024-04-03T05:19:11.642ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
17702024-04-03T05:19:11.642ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17712024-04-03T05:19:11.642ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17722024-04-03T05:19:11.642ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3544, tv_nsec: 323076153 } })
17732024-04-03T05:19:11.643ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:974 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
17742024-04-03T05:19:11.643ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 5844cfd7-028e-4204-ba41-2d4da75c3d1c dns_zone_name = oxide.test found_so_far = 1 version = 2
17752024-04-03T05:19:11.643ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
17762024-04-03T05:19:11.643ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-04-03 05:12:15.995755 UTC dns_group = external generation = 2
17772024-04-03T05:19:11.643ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 23.154488ms iteration = 9
17782024-04-03T05:19:11.644ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 1c2282ea-206d-4df3-bfca-a35a264cd464 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
17792024-04-03T05:19:11.644ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
17802024-04-03T05:19:11.644ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-04-03 05:12:15.091055 UTC dns_group = internal generation = 1
17812024-04-03T05:19:11.644ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 23.620295ms iteration = 9
17822024-04-03T05:19:11.645ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
17832024-04-03T05:19:11.645ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("9f24d5b2-898d-40ee-a459-c6576fb9d418.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c33ab12a-21de-454f-9b76-9a78bfc5c720.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c33ab12a-21de-454f-9b76-9a78bfc5c720.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3544, tv_nsec: 325869470 } })
17842024-04-03T05:19:11.646ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
17852024-04-03T05:19:11.646ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.264057ms iteration = 9
17862024-04-03T05:19:11.672ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:982 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 zone_address = fd00:1122:3344:101::2
17872024-04-03T05:19:11.672ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
17882024-04-03T05:19:11.673ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
17892024-04-03T05:19:11.673ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
17902024-04-03T05:19:11.673ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
17912024-04-03T05:19:11.673ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17922024-04-03T05:19:11.673ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
17932024-04-03T05:19:11.673ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
17942024-04-03T05:19:11.673ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
17952024-04-03T05:19:11.673ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
17962024-04-03T05:19:11.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
17972024-04-03T05:19:11.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17982024-04-03T05:19:11.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
17992024-04-03T05:19:11.673ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "6b0ba8c0-ef69-4a22-ad9f-41c88355766b", "content-length": "26", "date": "Wed, 03 Apr 2024 05:19:11 GMT"} })
18002024-04-03T05:19:11.673ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:987 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
18012024-04-03T05:19:11.675ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
18022024-04-03T05:19:11.677ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3544, tv_nsec: 357909337 } })
18032024-04-03T05:19:11.679ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18042024-04-03T05:19:11.679ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3544, tv_nsec: 359409687 } })
18052024-04-03T05:19:11.679ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:974
18062024-04-03T05:19:11.680ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:974
18072024-04-03T05:19:11.682ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
18082024-04-03T05:19:11.682ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 9.467556ms iteration = 16
18092024-04-03T05:19:11.682ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
18102024-04-03T05:19:11.682ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18112024-04-03T05:19:11.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18122024-04-03T05:19:11.684ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18132024-04-03T05:19:11.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18142024-04-03T05:19:11.699ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18152024-04-03T05:19:11.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18162024-04-03T05:19:11.701ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18172024-04-03T05:19:11.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18182024-04-03T05:19:11.702ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18192024-04-03T05:19:11.705ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 54b5d8ce-2f89-4588-80f9-2a71dfa53de1, lookup_type: ById(54b5d8ce-2f89-4588-80f9-2a71dfa53de1) } result = Ok(())
18202024-04-03T05:19:11.705ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18212024-04-03T05:19:11.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18222024-04-03T05:19:11.707ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 34.216686ms iteration = 16
18232024-04-03T05:19:11.780ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
18242024-04-03T05:19:11.780ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
18252024-04-03T05:19:11.780ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "1206c4c8-0419-42d8-95bb-6184ba9abd07", "content-length": "26", "date": "Wed, 03 Apr 2024 05:19:11 GMT"} })
18262024-04-03T05:19:11.780ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
18272024-04-03T05:19:11.782ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
18282024-04-03T05:19:11.881ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
18292024-04-03T05:19:11.881ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
18302024-04-03T05:19:11.881ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
18312024-04-03T05:19:11.882ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "1a350898-8d5a-477f-81f9-1299cf38ac67", "content-length": "26", "date": "Wed, 03 Apr 2024 05:19:11 GMT"} })
18322024-04-03T05:19:11.882ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
18332024-04-03T05:19:11.882ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "931a2844-e9ea-452a-ae55-15172675f14f", "content-length": "2", "date": "Wed, 03 Apr 2024 05:19:11 GMT"} })
18342024-04-03T05:19:11.884ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
18352024-04-03T05:19:11.884ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
18362024-04-03T05:19:11.884ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
18372024-04-03T05:19:11.886ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 213.21382ms iteration = 16
18382024-04-03T05:19:11.950ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
18392024-04-03T05:19:12.036ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 363.39002ms iteration = 16
18402024-04-03T05:19:12.042ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
18412024-04-03T05:19:12.101ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
18422024-04-03T05:19:12.102ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18432024-04-03T05:19:12.102ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "56047973-4810-440c-8753-a13510257d2f", "date": "Wed, 03 Apr 2024 05:19:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
18442024-04-03T05:19:12.102ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
18452024-04-03T05:19:12.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18462024-04-03T05:19:12.103ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "6b00fc68-232a-4f5e-8796-a8c648fdd09c", "date": "Wed, 03 Apr 2024 05:19:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
18472024-04-03T05:19:12.103ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
18482024-04-03T05:19:12.105ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:141 port = SwitchPort { id: 9abb0c53-3a74-441c-8a11-a2b0939760e0, rack_id: 38b3f10b-8962-43bd-8d32-8172eeecefb9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(0a16a58f-d519-4d96-87f3-b4294bb539f8) } rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
18492024-04-03T05:19:12.105ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18502024-04-03T05:19:12.105ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
18512024-04-03T05:19:12.105ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 154.220504ms iteration = 11
18522024-04-03T05:19:12.105ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18532024-04-03T05:19:12.111ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
18542024-04-03T05:19:12.111ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "eef14115-05ed-424e-8b84-0d7d5911c768", "content-length": "121", "date": "Wed, 03 Apr 2024 05:19:11 GMT"} })
18552024-04-03T05:19:12.112ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1418 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
18562024-04-03T05:19:12.112ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1445 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 0a16a58f-d519-4d96-87f3-b4294bb539f8
18572024-04-03T05:19:12.112ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:364 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
18582024-04-03T05:19:12.112ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:368 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
18592024-04-03T05:19:12.112ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:377 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
18602024-04-03T05:19:12.112ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:383 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
18612024-04-03T05:19:12.112ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:403 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
18622024-04-03T05:19:12.112ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18632024-04-03T05:19:12.112ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18642024-04-03T05:19:12.113ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
18652024-04-03T05:19:12.113ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b76c387f-984d-440d-b514-49e05d8b83d4", "content-length": "2", "date": "Wed, 03 Apr 2024 05:19:11 GMT"} })
18662024-04-03T05:19:12.113ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
18672024-04-03T05:19:12.113ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "577e77cf-c85c-4573-aac7-55ee15eacccd", "content-length": "52", "date": "Wed, 03 Apr 2024 05:19:11 GMT"} })
18682024-04-03T05:19:12.113ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:452 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
18692024-04-03T05:19:12.113ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
18702024-04-03T05:19:12.162ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "73927f37-f525-4871-8ff7-a99b10c21d07", "date": "Wed, 03 Apr 2024 05:19:12 GMT"} })
18712024-04-03T05:19:12.162ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:617 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch = Switch0
18722024-04-03T05:19:12.162ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
18732024-04-03T05:19:12.163ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "06b2e493-bf3c-4421-850b-3cc21313acc6", "content-length": "452", "date": "Wed, 03 Apr 2024 05:19:12 GMT"} })
18742024-04-03T05:19:12.163ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18752024-04-03T05:19:12.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18762024-04-03T05:19:12.166ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18772024-04-03T05:19:12.166ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18782024-04-03T05:19:12.168ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18792024-04-03T05:19:12.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18802024-04-03T05:19:12.170ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18812024-04-03T05:19:12.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18822024-04-03T05:19:12.177ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 541.009626ms iteration = 16
18832024-04-03T05:19:12.294ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
18842024-04-03T05:19:12.325ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
18852024-04-03T05:19:12.356ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
18862024-04-03T05:19:12.386ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
18872024-04-03T05:19:12.386ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 159 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 2c5e7b1c-875f-406d-9217-16afa4e3a1cd response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
18882024-04-03T05:19:12.386ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "974d6f64-7299-4943-8cf8-5a2f2011ec96", "date": "Wed, 03 Apr 2024 05:19:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
18892024-04-03T05:19:12.386ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
18902024-04-03T05:19:12.387ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "176efa08-cdef-45f4-ae64-b9e2f89791ea", "date": "Wed, 03 Apr 2024 05:19:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
18912024-04-03T05:19:12.387ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
18922024-04-03T05:19:12.389ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "955f1d7c-e0b4-45c4-8c87-a9cea43bb102", "date": "Wed, 03 Apr 2024 05:19:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
18932024-04-03T05:19:12.389ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
18942024-04-03T05:19:12.391ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
18952024-04-03T05:19:12.391ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 96.898751ms iteration = 11
18962024-04-03T05:19:21.621ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
18972024-04-03T05:19:21.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18982024-04-03T05:19:21.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18992024-04-03T05:19:21.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19002024-04-03T05:19:21.626ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
19012024-04-03T05:19:21.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19022024-04-03T05:19:21.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19032024-04-03T05:19:21.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19042024-04-03T05:19:21.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19052024-04-03T05:19:21.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19062024-04-03T05:19:21.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
19072024-04-03T05:19:21.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19082024-04-03T05:19:21.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19092024-04-03T05:19:21.636ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
19102024-04-03T05:19:21.636ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.566017ms iteration = 45
19112024-04-03T05:19:22.369ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 139 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 9bb7f01e-c651-4daf-8390-6b8fb2805cbc response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
19122024-04-03T05:19:31.621ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
19132024-04-03T05:19:31.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19142024-04-03T05:19:31.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19152024-04-03T05:19:31.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19162024-04-03T05:19:31.626ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
19172024-04-03T05:19:31.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19182024-04-03T05:19:31.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19192024-04-03T05:19:31.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19202024-04-03T05:19:31.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19212024-04-03T05:19:31.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19222024-04-03T05:19:31.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
19232024-04-03T05:19:31.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19242024-04-03T05:19:31.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19252024-04-03T05:19:31.636ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
19262024-04-03T05:19:31.636ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.335694ms iteration = 46
19272024-04-03T05:19:32.369ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 143 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 97e3d4cc-bdd9-4ace-934e-c6dc79d7055f response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
19282024-04-03T05:19:41.620ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
19292024-04-03T05:19:41.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19302024-04-03T05:19:41.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19312024-04-03T05:19:41.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19322024-04-03T05:19:41.625ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
19332024-04-03T05:19:41.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19342024-04-03T05:19:41.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19352024-04-03T05:19:41.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19362024-04-03T05:19:41.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19372024-04-03T05:19:41.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19382024-04-03T05:19:41.631ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
19392024-04-03T05:19:41.631ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19402024-04-03T05:19:41.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19412024-04-03T05:19:41.633ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
19422024-04-03T05:19:41.633ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
19432024-04-03T05:19:41.633ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
19442024-04-03T05:19:41.633ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 11.551\u{b5}s iteration = 17
19452024-04-03T05:19:41.636ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
19462024-04-03T05:19:41.636ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.463916ms iteration = 47
19472024-04-03T05:19:41.637ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 17 reason = Timeout
19482024-04-03T05:19:41.637ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19492024-04-03T05:19:41.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19502024-04-03T05:19:41.638ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19512024-04-03T05:19:41.640ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
19522024-04-03T05:19:41.640ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19532024-04-03T05:19:41.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19542024-04-03T05:19:41.642ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3574, tv_nsec: 322729478 } })
19552024-04-03T05:19:41.643ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:974 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
19562024-04-03T05:19:41.673ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:982 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 zone_address = fd00:1122:3344:101::2
19572024-04-03T05:19:41.673ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19582024-04-03T05:19:41.673ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "7d9ca140-024e-4fb6-bf1e-a2ca96261032", "content-length": "26", "date": "Wed, 03 Apr 2024 05:19:41 GMT"} })
19592024-04-03T05:19:41.673ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:987 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19602024-04-03T05:19:41.674ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
19612024-04-03T05:19:41.674ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
19622024-04-03T05:19:41.674ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
19632024-04-03T05:19:41.674ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
19642024-04-03T05:19:41.674ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
19652024-04-03T05:19:41.674ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19662024-04-03T05:19:41.674ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
19672024-04-03T05:19:41.674ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
19682024-04-03T05:19:41.674ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
19692024-04-03T05:19:41.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
19702024-04-03T05:19:41.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
19712024-04-03T05:19:41.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19722024-04-03T05:19:41.675ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19732024-04-03T05:19:41.676ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3574, tv_nsec: 356065722 } })
19742024-04-03T05:19:41.676ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
19752024-04-03T05:19:41.676ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.826068ms iteration = 17
19762024-04-03T05:19:41.676ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:974
19772024-04-03T05:19:41.677ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3574, tv_nsec: 356806561 } })
19782024-04-03T05:19:41.678ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:974
19792024-04-03T05:19:41.679ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
19802024-04-03T05:19:41.679ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19812024-04-03T05:19:41.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19822024-04-03T05:19:41.680ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19832024-04-03T05:19:41.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19842024-04-03T05:19:41.692ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19852024-04-03T05:19:41.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19862024-04-03T05:19:41.694ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19872024-04-03T05:19:41.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19882024-04-03T05:19:41.695ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19892024-04-03T05:19:41.698ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 54b5d8ce-2f89-4588-80f9-2a71dfa53de1, lookup_type: ById(54b5d8ce-2f89-4588-80f9-2a71dfa53de1) } result = Ok(())
19902024-04-03T05:19:41.698ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19912024-04-03T05:19:41.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19922024-04-03T05:19:41.699ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 25.623791ms iteration = 17
19932024-04-03T05:19:41.753ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
19942024-04-03T05:19:41.753ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19952024-04-03T05:19:41.753ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "8491bf75-50d0-4946-adcf-423cc73a6b57", "content-length": "26", "date": "Wed, 03 Apr 2024 05:19:41 GMT"} })
19962024-04-03T05:19:41.753ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19972024-04-03T05:19:41.756ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
19982024-04-03T05:19:41.759ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
19992024-04-03T05:19:41.759ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
20002024-04-03T05:19:41.759ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "278f5af2-8a51-46f0-94df-1a965fbefc22", "content-length": "26", "date": "Wed, 03 Apr 2024 05:19:41 GMT"} })
20012024-04-03T05:19:41.759ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
20022024-04-03T05:19:41.761ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
20032024-04-03T05:19:41.922ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20042024-04-03T05:19:41.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20052024-04-03T05:19:41.924ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:141 port = SwitchPort { id: 9abb0c53-3a74-441c-8a11-a2b0939760e0, rack_id: 38b3f10b-8962-43bd-8d32-8172eeecefb9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(0a16a58f-d519-4d96-87f3-b4294bb539f8) } rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
20062024-04-03T05:19:41.924ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20072024-04-03T05:19:41.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20082024-04-03T05:19:41.930ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
20092024-04-03T05:19:41.930ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "df35605e-f295-4c09-a130-d50f5453f073", "content-length": "121", "date": "Wed, 03 Apr 2024 05:19:41 GMT"} })
20102024-04-03T05:19:41.930ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1418 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
20112024-04-03T05:19:41.930ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1445 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 0a16a58f-d519-4d96-87f3-b4294bb539f8
20122024-04-03T05:19:41.931ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:364 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
20132024-04-03T05:19:41.931ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:368 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
20142024-04-03T05:19:41.931ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:377 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
20152024-04-03T05:19:41.931ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:383 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
20162024-04-03T05:19:41.931ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:403 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
20172024-04-03T05:19:41.931ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20182024-04-03T05:19:41.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20192024-04-03T05:19:41.932ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
20202024-04-03T05:19:41.932ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e5417089-487d-4b13-a683-64786d49f995", "content-length": "2", "date": "Wed, 03 Apr 2024 05:19:41 GMT"} })
20212024-04-03T05:19:41.933ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
20222024-04-03T05:19:41.933ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "ce56bf3e-f17a-4ec9-bfcc-c9b28f97bc7c", "content-length": "52", "date": "Wed, 03 Apr 2024 05:19:41 GMT"} })
20232024-04-03T05:19:41.933ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:452 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
20242024-04-03T05:19:41.933ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
20252024-04-03T05:19:41.943ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
20262024-04-03T05:19:41.943ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "5e1a6120-da5d-4c69-8dac-932f00899b5e", "content-length": "2", "date": "Wed, 03 Apr 2024 05:19:41 GMT"} })
20272024-04-03T05:19:41.944ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 270.353252ms iteration = 17
20282024-04-03T05:19:41.944ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
20292024-04-03T05:19:41.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
20302024-04-03T05:19:41.946ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 272.267657ms iteration = 17
20312024-04-03T05:19:42.028ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "b7a85c78-a499-43b8-b70a-6780c58dc071", "date": "Wed, 03 Apr 2024 05:19:41 GMT"} })
20322024-04-03T05:19:42.028ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:617 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch = Switch0
20332024-04-03T05:19:42.028ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
20342024-04-03T05:19:42.032ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b624bea2-62d4-49b0-b22d-ee8b0ab25b13", "content-length": "452", "date": "Wed, 03 Apr 2024 05:19:41 GMT"} })
20352024-04-03T05:19:42.032ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20362024-04-03T05:19:42.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20372024-04-03T05:19:42.036ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20382024-04-03T05:19:42.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20392024-04-03T05:19:42.038ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20402024-04-03T05:19:42.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20412024-04-03T05:19:42.040ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20422024-04-03T05:19:42.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20432024-04-03T05:19:42.046ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 409.536156ms iteration = 17
20442024-04-03T05:19:42.369ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 116 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 925f88f3-82a7-4c91-bbff-f5b54903808c response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
20452024-04-03T05:19:51.620ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
20462024-04-03T05:19:51.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20472024-04-03T05:19:51.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20482024-04-03T05:19:51.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20492024-04-03T05:19:51.626ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
20502024-04-03T05:19:51.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20512024-04-03T05:19:51.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20522024-04-03T05:19:51.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20532024-04-03T05:19:51.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20542024-04-03T05:19:51.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20552024-04-03T05:19:51.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
20562024-04-03T05:19:51.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20572024-04-03T05:19:51.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20582024-04-03T05:19:51.637ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
20592024-04-03T05:19:51.637ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.252176ms iteration = 48
20602024-04-03T05:19:52.370ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 130 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 274d4a83-2168-4067-b463-a0cc3c3428c7 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
20612024-04-03T05:20:01.620ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
20622024-04-03T05:20:01.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20632024-04-03T05:20:01.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20642024-04-03T05:20:01.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20652024-04-03T05:20:01.625ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
20662024-04-03T05:20:01.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20672024-04-03T05:20:01.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20682024-04-03T05:20:01.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20692024-04-03T05:20:01.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20702024-04-03T05:20:01.634ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20712024-04-03T05:20:01.636ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
20722024-04-03T05:20:01.636ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20732024-04-03T05:20:01.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20742024-04-03T05:20:01.641ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
20752024-04-03T05:20:01.641ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.175149ms iteration = 49
20762024-04-03T05:20:02.370ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 141 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 3efec457-71ea-4b23-8c52-7d1131613009 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
20772024-04-03T05:20:11.621ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
20782024-04-03T05:20:11.621ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 10 reason = Timeout
20792024-04-03T05:20:11.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20802024-04-03T05:20:11.621ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 10 reason = Timeout
20812024-04-03T05:20:11.621ZWARNnexus (BackgroundTasks): Metric producer GC: statically disabled pending omicron#5284 background_task = metrics_producer_gc file = nexus/src/app/background/metrics_producer_gc.rs:43
20822024-04-03T05:20:11.621ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20832024-04-03T05:20:11.621ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 15.241\u{b5}s iteration = 10
20842024-04-03T05:20:11.621ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 10 reason = Timeout
20852024-04-03T05:20:11.621ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
20862024-04-03T05:20:11.621ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 10 reason = Timeout
20872024-04-03T05:20:11.621ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20882024-04-03T05:20:11.621ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 10 reason = Timeout
20892024-04-03T05:20:11.621ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
20902024-04-03T05:20:11.621ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20912024-04-03T05:20:11.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20922024-04-03T05:20:11.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20932024-04-03T05:20:11.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20942024-04-03T05:20:11.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20952024-04-03T05:20:11.622ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("174b6905-67c4-445e-8ee0-92f9d656bfce.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1b4f3db-685d-4e05-94cb-90ed84711941.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a2b50f43-9fdf-42d4-88bc-53143a838677.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("a2b50f43-9fdf-42d4-88bc-53143a838677.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 3604, tv_nsec: 301759366 } })
20962024-04-03T05:20:11.623ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20972024-04-03T05:20:11.623ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20982024-04-03T05:20:11.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20992024-04-03T05:20:11.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21002024-04-03T05:20:11.623ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
21012024-04-03T05:20:11.623ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 2.452602ms iteration = 10
21022024-04-03T05:20:11.624ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21032024-04-03T05:20:11.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21042024-04-03T05:20:11.624ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21052024-04-03T05:20:11.624ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21062024-04-03T05:20:11.626ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
21072024-04-03T05:20:11.626ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21082024-04-03T05:20:11.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21092024-04-03T05:20:11.629ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21102024-04-03T05:20:11.629ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
21112024-04-03T05:20:11.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21122024-04-03T05:20:11.629ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
21132024-04-03T05:20:11.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21142024-04-03T05:20:11.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21152024-04-03T05:20:11.629ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
21162024-04-03T05:20:11.630ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
21172024-04-03T05:20:11.630ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
21182024-04-03T05:20:11.630ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
21192024-04-03T05:20:11.630ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21202024-04-03T05:20:11.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21212024-04-03T05:20:11.630ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21222024-04-03T05:20:11.630ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
21232024-04-03T05:20:11.630ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
21242024-04-03T05:20:11.630ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
21252024-04-03T05:20:11.630ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21262024-04-03T05:20:11.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21272024-04-03T05:20:11.630ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21282024-04-03T05:20:11.631ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 11 reason = Timeout
21292024-04-03T05:20:11.631ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:65 reason = blueprint disabled target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e
21302024-04-03T05:20:11.631ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 12.08\u{b5}s iteration = 11
21312024-04-03T05:20:11.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21322024-04-03T05:20:11.631ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21332024-04-03T05:20:11.631ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21342024-04-03T05:20:11.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21352024-04-03T05:20:11.633ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
21362024-04-03T05:20:11.633ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
21372024-04-03T05:20:11.633ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
21382024-04-03T05:20:11.633ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 9.6\u{b5}s iteration = 18
21392024-04-03T05:20:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
21402024-04-03T05:20:11.634ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21412024-04-03T05:20:11.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21422024-04-03T05:20:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
21432024-04-03T05:20:11.635ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21442024-04-03T05:20:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21452024-04-03T05:20:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
21462024-04-03T05:20:11.635ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21472024-04-03T05:20:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
21482024-04-03T05:20:11.635ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21492024-04-03T05:20:11.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21502024-04-03T05:20:11.636ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21512024-04-03T05:20:11.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21522024-04-03T05:20:11.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21532024-04-03T05:20:11.637ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
21542024-04-03T05:20:11.637ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 1c2282ea-206d-4df3-bfca-a35a264cd464 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
21552024-04-03T05:20:11.637ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21562024-04-03T05:20:11.637ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21572024-04-03T05:20:11.637ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
21582024-04-03T05:20:11.637ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 5844cfd7-028e-4204-ba41-2d4da75c3d1c dns_zone_name = oxide.test found_so_far = 0 version = 2
21592024-04-03T05:20:11.637ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21602024-04-03T05:20:11.637ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 18 reason = Timeout
21612024-04-03T05:20:11.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21622024-04-03T05:20:11.637ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21632024-04-03T05:20:11.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21642024-04-03T05:20:11.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21652024-04-03T05:20:11.638ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21662024-04-03T05:20:11.638ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21672024-04-03T05:20:11.639ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21682024-04-03T05:20:11.639ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
21692024-04-03T05:20:11.639ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21702024-04-03T05:20:11.639ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
21712024-04-03T05:20:11.639ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.507417ms iteration = 50
21722024-04-03T05:20:11.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21732024-04-03T05:20:11.641ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
21742024-04-03T05:20:11.641ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21752024-04-03T05:20:11.642ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21762024-04-03T05:20:11.642ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
21772024-04-03T05:20:11.642ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21782024-04-03T05:20:11.642ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21792024-04-03T05:20:11.642ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-04-03T05:12:11.760059Z, time_modified: 2024-04-03T05:12:11.760059Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: d254228d-6b17-46e4-bc06-e18669dfc21c, db_silo: Silo { identity: SiloIdentity { id: d254228d-6b17-46e4-bc06-e18669dfc21c, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-04-03T05:12:15.962604Z, time_modified: 2024-04-03T05:12:15.962604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "096b197d9f10cd1b196c898f82b2d3451036c6b5bc8fff5b9af57c633900dba5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: d254228d-6b17-46e4-bc06-e18669dfc21c, db_silo: Silo { identity: SiloIdentity { id: d254228d-6b17-46e4-bc06-e18669dfc21c, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-04-03T05:12:15.962604Z, time_modified: 2024-04-03T05:12:15.962604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "096b197d9f10cd1b196c898f82b2d3451036c6b5bc8fff5b9af57c633900dba5" }] }) }
21802024-04-03T05:20:11.642ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 21.361884ms iteration = 10
21812024-04-03T05:20:11.642ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
21822024-04-03T05:20:11.643ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21832024-04-03T05:20:11.643ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21842024-04-03T05:20:11.643ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3604, tv_nsec: 322922908 } })
21852024-04-03T05:20:11.644ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 5844cfd7-028e-4204-ba41-2d4da75c3d1c dns_zone_name = oxide.test found_so_far = 1 version = 2
21862024-04-03T05:20:11.644ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
21872024-04-03T05:20:11.644ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-04-03 05:12:15.995755 UTC dns_group = external generation = 2
21882024-04-03T05:20:11.644ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 22.53154ms iteration = 10
21892024-04-03T05:20:11.644ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:974 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
21902024-04-03T05:20:11.644ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 1c2282ea-206d-4df3-bfca-a35a264cd464 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
21912024-04-03T05:20:11.644ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
21922024-04-03T05:20:11.644ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-04-03 05:12:15.091055 UTC dns_group = internal generation = 1
21932024-04-03T05:20:11.644ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 23.29108ms iteration = 10
21942024-04-03T05:20:11.646ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 10 reason = Timeout
21952024-04-03T05:20:11.647ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("9f24d5b2-898d-40ee-a459-c6576fb9d418.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c33ab12a-21de-454f-9b76-9a78bfc5c720.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c33ab12a-21de-454f-9b76-9a78bfc5c720.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3604, tv_nsec: 327113384 } })
21962024-04-03T05:20:11.648ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
21972024-04-03T05:20:11.648ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 2.018027ms iteration = 10
21982024-04-03T05:20:11.674ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:982 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 zone_address = fd00:1122:3344:101::2
21992024-04-03T05:20:11.674ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
22002024-04-03T05:20:11.674ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
22012024-04-03T05:20:11.674ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
22022024-04-03T05:20:11.674ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
22032024-04-03T05:20:11.674ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22042024-04-03T05:20:11.674ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
22052024-04-03T05:20:11.674ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
22062024-04-03T05:20:11.674ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
22072024-04-03T05:20:11.674ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
22082024-04-03T05:20:11.674ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "73b3117c-e86e-42de-94f5-c8b16c0f42bb", "content-length": "26", "date": "Wed, 03 Apr 2024 05:20:11 GMT"} })
22092024-04-03T05:20:11.675ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:987 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
22102024-04-03T05:20:11.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
22112024-04-03T05:20:11.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
22122024-04-03T05:20:11.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22132024-04-03T05:20:11.676ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3604, tv_nsec: 355739835 } })
22142024-04-03T05:20:11.676ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22152024-04-03T05:20:11.677ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3604, tv_nsec: 356476915 } })
22162024-04-03T05:20:11.677ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
22172024-04-03T05:20:11.677ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.362191ms iteration = 18
22182024-04-03T05:20:11.677ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:974
22192024-04-03T05:20:11.677ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
22202024-04-03T05:20:11.677ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:974
22212024-04-03T05:20:11.679ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
22222024-04-03T05:20:11.679ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22232024-04-03T05:20:11.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22242024-04-03T05:20:11.680ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22252024-04-03T05:20:11.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22262024-04-03T05:20:11.693ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22272024-04-03T05:20:11.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22282024-04-03T05:20:11.694ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22292024-04-03T05:20:11.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22302024-04-03T05:20:11.696ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22312024-04-03T05:20:11.699ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 54b5d8ce-2f89-4588-80f9-2a71dfa53de1, lookup_type: ById(54b5d8ce-2f89-4588-80f9-2a71dfa53de1) } result = Ok(())
22322024-04-03T05:20:11.699ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22332024-04-03T05:20:11.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22342024-04-03T05:20:11.700ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 25.694252ms iteration = 18
22352024-04-03T05:20:11.771ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
22362024-04-03T05:20:11.771ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
22372024-04-03T05:20:11.771ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "67085b54-34b0-4ba4-a92a-1577de3ddb37", "content-length": "26", "date": "Wed, 03 Apr 2024 05:20:11 GMT"} })
22382024-04-03T05:20:11.771ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
22392024-04-03T05:20:11.772ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
22402024-04-03T05:20:11.772ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
22412024-04-03T05:20:11.773ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
22422024-04-03T05:20:11.774ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "94086175-17e0-479b-9d79-f92530d1de92", "content-length": "26", "date": "Wed, 03 Apr 2024 05:20:11 GMT"} })
22432024-04-03T05:20:11.774ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
22442024-04-03T05:20:11.775ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
22452024-04-03T05:20:11.951ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 12 reason = Timeout
22462024-04-03T05:20:11.965ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
22472024-04-03T05:20:11.965ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "c3f7d3c6-ffb8-43c6-b77c-fa8f981a7efd", "content-length": "2", "date": "Wed, 03 Apr 2024 05:20:11 GMT"} })
22482024-04-03T05:20:11.966ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
22492024-04-03T05:20:11.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
22502024-04-03T05:20:11.968ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 293.566642ms iteration = 18
22512024-04-03T05:20:11.969ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22522024-04-03T05:20:11.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22532024-04-03T05:20:11.976ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 301.391745ms iteration = 18
22542024-04-03T05:20:11.978ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:141 port = SwitchPort { id: 9abb0c53-3a74-441c-8a11-a2b0939760e0, rack_id: 38b3f10b-8962-43bd-8d32-8172eeecefb9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(0a16a58f-d519-4d96-87f3-b4294bb539f8) } rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
22552024-04-03T05:20:11.978ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22562024-04-03T05:20:11.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22572024-04-03T05:20:11.984ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
22582024-04-03T05:20:11.984ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "3b863104-6560-429e-b338-42b18d72f70c", "content-length": "121", "date": "Wed, 03 Apr 2024 05:20:11 GMT"} })
22592024-04-03T05:20:11.984ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1418 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
22602024-04-03T05:20:11.984ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1445 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 0a16a58f-d519-4d96-87f3-b4294bb539f8
22612024-04-03T05:20:11.985ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:364 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
22622024-04-03T05:20:11.985ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:368 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
22632024-04-03T05:20:11.985ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:377 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
22642024-04-03T05:20:11.985ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:383 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
22652024-04-03T05:20:11.985ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:403 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
22662024-04-03T05:20:11.985ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22672024-04-03T05:20:11.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22682024-04-03T05:20:11.986ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
22692024-04-03T05:20:11.986ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "8a73d04d-e64d-49be-9b30-af34f2541622", "content-length": "2", "date": "Wed, 03 Apr 2024 05:20:11 GMT"} })
22702024-04-03T05:20:11.986ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
22712024-04-03T05:20:11.987ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d8395a42-e6bc-45d3-a69c-25ad189118eb", "content-length": "52", "date": "Wed, 03 Apr 2024 05:20:11 GMT"} })
22722024-04-03T05:20:11.987ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:452 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
22732024-04-03T05:20:11.987ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
22742024-04-03T05:20:11.996ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
22752024-04-03T05:20:12.026ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
22762024-04-03T05:20:12.027ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "037647d5-724c-451b-ad87-0f61b7bb4534", "date": "Wed, 03 Apr 2024 05:20:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
22772024-04-03T05:20:12.027ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
22782024-04-03T05:20:12.028ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "9da37899-a8f7-4b9a-97f0-377f24ad030d", "date": "Wed, 03 Apr 2024 05:20:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
22792024-04-03T05:20:12.028ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
22802024-04-03T05:20:12.029ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
22812024-04-03T05:20:12.029ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 77.378351ms iteration = 12
22822024-04-03T05:20:12.036ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "a7d5b845-49b1-4e10-a866-5a375706ed30", "date": "Wed, 03 Apr 2024 05:20:11 GMT"} })
22832024-04-03T05:20:12.036ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:617 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch = Switch0
22842024-04-03T05:20:12.036ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
22852024-04-03T05:20:12.037ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "14a79732-cb96-423a-8f5c-b51c3a0292b2", "content-length": "452", "date": "Wed, 03 Apr 2024 05:20:11 GMT"} })
22862024-04-03T05:20:12.037ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22872024-04-03T05:20:12.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22882024-04-03T05:20:12.041ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22892024-04-03T05:20:12.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22902024-04-03T05:20:12.043ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22912024-04-03T05:20:12.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22922024-04-03T05:20:12.045ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22932024-04-03T05:20:12.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22942024-04-03T05:20:12.053ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 415.889321ms iteration = 18
22952024-04-03T05:20:12.295ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 12 reason = Timeout
22962024-04-03T05:20:12.325ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
22972024-04-03T05:20:12.355ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
22982024-04-03T05:20:12.385ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
22992024-04-03T05:20:12.385ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 152 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 990f5a6e-da86-4d74-8a7a-5f629d27c630 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
23002024-04-03T05:20:12.385ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "9ed77608-cf47-4888-8d07-d3f8bb50d544", "date": "Wed, 03 Apr 2024 05:20:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
23012024-04-03T05:20:12.385ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
23022024-04-03T05:20:12.386ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "429c2d69-0b95-4a97-8e78-706cd125f92c", "date": "Wed, 03 Apr 2024 05:20:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
23032024-04-03T05:20:12.386ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
23042024-04-03T05:20:12.388ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "4fd7e83b-cb5b-41a9-9251-e32344f17ca7", "date": "Wed, 03 Apr 2024 05:20:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
23052024-04-03T05:20:12.388ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
23062024-04-03T05:20:12.390ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
23072024-04-03T05:20:12.390ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 94.53477ms iteration = 12
23082024-04-03T05:20:21.621ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
23092024-04-03T05:20:21.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23102024-04-03T05:20:21.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23112024-04-03T05:20:21.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23122024-04-03T05:20:21.626ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
23132024-04-03T05:20:21.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23142024-04-03T05:20:21.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23152024-04-03T05:20:21.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23162024-04-03T05:20:21.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23172024-04-03T05:20:21.630ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23182024-04-03T05:20:21.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
23192024-04-03T05:20:21.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23202024-04-03T05:20:21.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23212024-04-03T05:20:21.636ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
23222024-04-03T05:20:21.636ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.554937ms iteration = 51
23232024-04-03T05:20:22.370ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 118 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = b59db2c4-d272-4e62-932b-198256d7379c response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
23242024-04-03T05:20:31.621ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 52 reason = Timeout
23252024-04-03T05:20:31.622ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23262024-04-03T05:20:31.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23272024-04-03T05:20:31.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23282024-04-03T05:20:31.627ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
23292024-04-03T05:20:31.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23302024-04-03T05:20:31.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23312024-04-03T05:20:31.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23322024-04-03T05:20:31.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23332024-04-03T05:20:31.630ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23342024-04-03T05:20:31.633ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
23352024-04-03T05:20:31.633ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23362024-04-03T05:20:31.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23372024-04-03T05:20:31.637ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
23382024-04-03T05:20:31.637ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.057684ms iteration = 52
23392024-04-03T05:20:32.370ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 132 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 33281280-2bb9-4bb3-98eb-cb3fc7b65df9 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
23402024-04-03T05:20:41.621ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 53 reason = Timeout
23412024-04-03T05:20:41.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23422024-04-03T05:20:41.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23432024-04-03T05:20:41.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23442024-04-03T05:20:41.626ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
23452024-04-03T05:20:41.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23462024-04-03T05:20:41.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23472024-04-03T05:20:41.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23482024-04-03T05:20:41.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23492024-04-03T05:20:41.630ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23502024-04-03T05:20:41.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
23512024-04-03T05:20:41.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23522024-04-03T05:20:41.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23532024-04-03T05:20:41.633ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 19 reason = Timeout
23542024-04-03T05:20:41.633ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
23552024-04-03T05:20:41.633ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
23562024-04-03T05:20:41.633ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 9.83\u{b5}s iteration = 19
23572024-04-03T05:20:41.636ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
23582024-04-03T05:20:41.636ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.094461ms iteration = 53
23592024-04-03T05:20:41.637ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 19 reason = Timeout
23602024-04-03T05:20:41.637ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23612024-04-03T05:20:41.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23622024-04-03T05:20:41.639ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23632024-04-03T05:20:41.640ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
23642024-04-03T05:20:41.641ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23652024-04-03T05:20:41.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23662024-04-03T05:20:41.643ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3634, tv_nsec: 322078666 } })
23672024-04-03T05:20:41.643ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:974 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
23682024-04-03T05:20:41.673ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:982 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 zone_address = fd00:1122:3344:101::2
23692024-04-03T05:20:41.674ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
23702024-04-03T05:20:41.674ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2153ca2e-79a0-4d9e-80d6-d260de0537e1", "content-length": "26", "date": "Wed, 03 Apr 2024 05:20:41 GMT"} })
23712024-04-03T05:20:41.674ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:987 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
23722024-04-03T05:20:41.674ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
23732024-04-03T05:20:41.674ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 19 reason = Timeout
23742024-04-03T05:20:41.675ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
23752024-04-03T05:20:41.675ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 19 reason = Timeout
23762024-04-03T05:20:41.675ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 19 reason = Timeout
23772024-04-03T05:20:41.675ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 19 reason = Timeout
23782024-04-03T05:20:41.675ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
23792024-04-03T05:20:41.675ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
23802024-04-03T05:20:41.675ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23812024-04-03T05:20:41.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
23822024-04-03T05:20:41.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
23832024-04-03T05:20:41.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23842024-04-03T05:20:41.676ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23852024-04-03T05:20:41.677ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3634, tv_nsec: 356414143 } })
23862024-04-03T05:20:41.677ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
23872024-04-03T05:20:41.677ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.585205ms iteration = 19
23882024-04-03T05:20:41.677ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3634, tv_nsec: 356548825 } })
23892024-04-03T05:20:41.678ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:974
23902024-04-03T05:20:41.679ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
23912024-04-03T05:20:41.679ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23922024-04-03T05:20:41.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23932024-04-03T05:20:41.679ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:974
23942024-04-03T05:20:41.680ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23952024-04-03T05:20:41.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23962024-04-03T05:20:41.698ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23972024-04-03T05:20:41.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23982024-04-03T05:20:41.699ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23992024-04-03T05:20:41.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24002024-04-03T05:20:41.701ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24012024-04-03T05:20:41.704ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 54b5d8ce-2f89-4588-80f9-2a71dfa53de1, lookup_type: ById(54b5d8ce-2f89-4588-80f9-2a71dfa53de1) } result = Ok(())
24022024-04-03T05:20:41.704ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24032024-04-03T05:20:41.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24042024-04-03T05:20:41.705ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 30.558377ms iteration = 19
24052024-04-03T05:20:41.754ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
24062024-04-03T05:20:41.754ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
24072024-04-03T05:20:41.755ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "7b42a299-2770-4ed6-a9ef-092ae9da643f", "content-length": "26", "date": "Wed, 03 Apr 2024 05:20:41 GMT"} })
24082024-04-03T05:20:41.755ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
24092024-04-03T05:20:41.756ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
24102024-04-03T05:20:41.762ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
24112024-04-03T05:20:41.762ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
24122024-04-03T05:20:41.762ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "8d129f87-a6f4-4528-b5fc-b5c8a1642b81", "content-length": "26", "date": "Wed, 03 Apr 2024 05:20:41 GMT"} })
24132024-04-03T05:20:41.762ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
24142024-04-03T05:20:41.764ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
24152024-04-03T05:20:41.872ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 197.381429ms iteration = 19
24162024-04-03T05:20:41.932ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
24172024-04-03T05:20:41.932ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e78f8074-703e-4725-b95d-6d98021e4f57", "content-length": "2", "date": "Wed, 03 Apr 2024 05:20:41 GMT"} })
24182024-04-03T05:20:41.933ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
24192024-04-03T05:20:41.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
24202024-04-03T05:20:41.935ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 260.140596ms iteration = 19
24212024-04-03T05:20:41.958ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24222024-04-03T05:20:41.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24232024-04-03T05:20:41.960ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:141 port = SwitchPort { id: 9abb0c53-3a74-441c-8a11-a2b0939760e0, rack_id: 38b3f10b-8962-43bd-8d32-8172eeecefb9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(0a16a58f-d519-4d96-87f3-b4294bb539f8) } rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
24242024-04-03T05:20:41.960ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24252024-04-03T05:20:41.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24262024-04-03T05:20:41.966ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
24272024-04-03T05:20:41.966ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "da035862-c054-4f88-837f-bfdea4115c42", "content-length": "121", "date": "Wed, 03 Apr 2024 05:20:41 GMT"} })
24282024-04-03T05:20:41.966ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1418 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
24292024-04-03T05:20:41.966ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1445 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 0a16a58f-d519-4d96-87f3-b4294bb539f8
24302024-04-03T05:20:41.967ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:364 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
24312024-04-03T05:20:41.967ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:368 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
24322024-04-03T05:20:41.967ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:377 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
24332024-04-03T05:20:41.967ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:383 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
24342024-04-03T05:20:41.967ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:403 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
24352024-04-03T05:20:41.967ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24362024-04-03T05:20:41.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24372024-04-03T05:20:41.968ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
24382024-04-03T05:20:41.968ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d379ca30-f3b6-4bd6-aa59-c1f07840d8b9", "content-length": "2", "date": "Wed, 03 Apr 2024 05:20:41 GMT"} })
24392024-04-03T05:20:41.968ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
24402024-04-03T05:20:41.968ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "dc787029-6d23-498b-88c1-de123f802126", "content-length": "52", "date": "Wed, 03 Apr 2024 05:20:41 GMT"} })
24412024-04-03T05:20:41.968ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:452 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
24422024-04-03T05:20:41.968ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
24432024-04-03T05:20:42.018ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "a80b1c8d-0944-4dd3-8600-5ec99aa98132", "date": "Wed, 03 Apr 2024 05:20:41 GMT"} })
24442024-04-03T05:20:42.018ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:617 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch = Switch0
24452024-04-03T05:20:42.018ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
24462024-04-03T05:20:42.022ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "ebb5ab44-a8fb-4ff9-88da-1efca6a336b0", "content-length": "452", "date": "Wed, 03 Apr 2024 05:20:42 GMT"} })
24472024-04-03T05:20:42.022ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24482024-04-03T05:20:42.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24492024-04-03T05:20:42.025ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24502024-04-03T05:20:42.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24512024-04-03T05:20:42.027ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24522024-04-03T05:20:42.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24532024-04-03T05:20:42.028ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24542024-04-03T05:20:42.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24552024-04-03T05:20:42.036ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 398.411678ms iteration = 19
24562024-04-03T05:20:42.371ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 123 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 381d59e4-13f5-4651-ad94-7e30f2d3c4ed response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
24572024-04-03T05:20:51.621ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 54 reason = Timeout
24582024-04-03T05:20:51.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24592024-04-03T05:20:51.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24602024-04-03T05:20:51.623ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24612024-04-03T05:20:51.626ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
24622024-04-03T05:20:51.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24632024-04-03T05:20:51.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24642024-04-03T05:20:51.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24652024-04-03T05:20:51.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24662024-04-03T05:20:51.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24672024-04-03T05:20:51.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
24682024-04-03T05:20:51.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24692024-04-03T05:20:51.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24702024-04-03T05:20:51.636ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
24712024-04-03T05:20:51.636ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 14.879978ms iteration = 54
24722024-04-03T05:20:52.371ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 174 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 0d00b0b4-b620-4a9d-9859-14e356bd3fb2 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
24732024-04-03T05:21:01.622ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 55 reason = Timeout
24742024-04-03T05:21:01.622ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24752024-04-03T05:21:01.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24762024-04-03T05:21:01.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24772024-04-03T05:21:01.627ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
24782024-04-03T05:21:01.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24792024-04-03T05:21:01.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24802024-04-03T05:21:01.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24812024-04-03T05:21:01.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24822024-04-03T05:21:01.630ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24832024-04-03T05:21:01.633ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
24842024-04-03T05:21:01.633ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24852024-04-03T05:21:01.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24862024-04-03T05:21:01.637ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
24872024-04-03T05:21:01.637ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 14.920118ms iteration = 55
24882024-04-03T05:21:02.371ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 203 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = b319cc7f-63fc-4b70-8a5d-010900532035 response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
24892024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 56 reason = Timeout
24902024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24912024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 11 reason = Timeout
24922024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
24932024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 11 reason = Timeout
24942024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
24952024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
24962024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 11 reason = Timeout
24972024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
24982024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
24992024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 11 reason = Timeout
25002024-04-03T05:21:11.622ZWARNnexus (BackgroundTasks): Metric producer GC: statically disabled pending omicron#5284 background_task = metrics_producer_gc file = nexus/src/app/background/metrics_producer_gc.rs:43
25012024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 15.66\u{b5}s iteration = 11
25022024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 11 reason = Timeout
25032024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25042024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
25052024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
25062024-04-03T05:21:11.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
25072024-04-03T05:21:11.623ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("174b6905-67c4-445e-8ee0-92f9d656bfce.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1b4f3db-685d-4e05-94cb-90ed84711941.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a2b50f43-9fdf-42d4-88bc-53143a838677.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("a2b50f43-9fdf-42d4-88bc-53143a838677.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 3664, tv_nsec: 301693744 } })
25082024-04-03T05:21:11.623ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
25092024-04-03T05:21:11.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
25102024-04-03T05:21:11.624ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
25112024-04-03T05:21:11.624ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.854335ms iteration = 11
25122024-04-03T05:21:11.624ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
25132024-04-03T05:21:11.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25142024-04-03T05:21:11.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
25152024-04-03T05:21:11.625ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25162024-04-03T05:21:11.625ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25172024-04-03T05:21:11.626ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25182024-04-03T05:21:11.628ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
25192024-04-03T05:21:11.628ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
25202024-04-03T05:21:11.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
25212024-04-03T05:21:11.628ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25222024-04-03T05:21:11.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25232024-04-03T05:21:11.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25242024-04-03T05:21:11.630ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
25252024-04-03T05:21:11.630ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
25262024-04-03T05:21:11.630ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
25272024-04-03T05:21:11.630ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25282024-04-03T05:21:11.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
25292024-04-03T05:21:11.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25302024-04-03T05:21:11.630ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
25312024-04-03T05:21:11.630ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
25322024-04-03T05:21:11.630ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
25332024-04-03T05:21:11.630ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
25342024-04-03T05:21:11.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
25352024-04-03T05:21:11.631ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
25362024-04-03T05:21:11.631ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
25372024-04-03T05:21:11.631ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
25382024-04-03T05:21:11.631ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
25392024-04-03T05:21:11.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
25402024-04-03T05:21:11.631ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25412024-04-03T05:21:11.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25422024-04-03T05:21:11.632ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25432024-04-03T05:21:11.632ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 12 reason = Timeout
25442024-04-03T05:21:11.632ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25452024-04-03T05:21:11.632ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:65 reason = blueprint disabled target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e
25462024-04-03T05:21:11.632ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 11.57\u{b5}s iteration = 12
25472024-04-03T05:21:11.634ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 20 reason = Timeout
25482024-04-03T05:21:11.634ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
25492024-04-03T05:21:11.634ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
25502024-04-03T05:21:11.634ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 11.09\u{b5}s iteration = 20
25512024-04-03T05:21:11.635ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
25522024-04-03T05:21:11.635ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
25532024-04-03T05:21:11.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
25542024-04-03T05:21:11.636ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
25552024-04-03T05:21:11.636ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
25562024-04-03T05:21:11.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
25572024-04-03T05:21:11.637ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f5527b9-cd4c-45e9-8883-5bd926c9178e, lookup_type: ById(0f5527b9-cd4c-45e9-8883-5bd926c9178e) } result = Ok(())
25582024-04-03T05:21:11.637ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25592024-04-03T05:21:11.637ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
25602024-04-03T05:21:11.637ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
25612024-04-03T05:21:11.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
25622024-04-03T05:21:11.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25632024-04-03T05:21:11.637ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
25642024-04-03T05:21:11.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
25652024-04-03T05:21:11.638ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 20 reason = Timeout
25662024-04-03T05:21:11.638ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25672024-04-03T05:21:11.638ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
25682024-04-03T05:21:11.638ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 1c2282ea-206d-4df3-bfca-a35a264cd464 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
25692024-04-03T05:21:11.638ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
25702024-04-03T05:21:11.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25712024-04-03T05:21:11.638ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
25722024-04-03T05:21:11.638ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 5844cfd7-028e-4204-ba41-2d4da75c3d1c dns_zone_name = oxide.test found_so_far = 0 version = 2
25732024-04-03T05:21:11.638ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
25742024-04-03T05:21:11.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
25752024-04-03T05:21:11.638ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25762024-04-03T05:21:11.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
25772024-04-03T05:21:11.639ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25782024-04-03T05:21:11.639ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25792024-04-03T05:21:11.640ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25802024-04-03T05:21:11.641ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
25812024-04-03T05:21:11.641ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
25822024-04-03T05:21:11.641ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e original_time_created = 2024-04-03 05:12:01.079669 UTC target_id = 0f5527b9-cd4c-45e9-8883-5bd926c9178e time_created = 2024-04-03 05:12:01.079669 UTC
25832024-04-03T05:21:11.641ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.739503ms iteration = 56
25842024-04-03T05:21:11.642ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
25852024-04-03T05:21:11.643ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
25862024-04-03T05:21:11.643ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25872024-04-03T05:21:11.643ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25882024-04-03T05:21:11.644ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
25892024-04-03T05:21:11.644ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
25902024-04-03T05:21:11.644ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
25912024-04-03T05:21:11.644ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
25922024-04-03T05:21:11.644ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
25932024-04-03T05:21:11.644ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-04-03T05:12:11.760059Z, time_modified: 2024-04-03T05:12:11.760059Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: d254228d-6b17-46e4-bc06-e18669dfc21c, db_silo: Silo { identity: SiloIdentity { id: d254228d-6b17-46e4-bc06-e18669dfc21c, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-04-03T05:12:15.962604Z, time_modified: 2024-04-03T05:12:15.962604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "096b197d9f10cd1b196c898f82b2d3451036c6b5bc8fff5b9af57c633900dba5" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: d254228d-6b17-46e4-bc06-e18669dfc21c, db_silo: Silo { identity: SiloIdentity { id: d254228d-6b17-46e4-bc06-e18669dfc21c, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-04-03T05:12:15.962604Z, time_modified: 2024-04-03T05:12:15.962604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "096b197d9f10cd1b196c898f82b2d3451036c6b5bc8fff5b9af57c633900dba5" }] }) }
25942024-04-03T05:21:11.644ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 22.215426ms iteration = 11
25952024-04-03T05:21:11.644ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
25962024-04-03T05:21:11.645ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3664, tv_nsec: 323650057 } })
25972024-04-03T05:21:11.645ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 5844cfd7-028e-4204-ba41-2d4da75c3d1c dns_zone_name = oxide.test found_so_far = 1 version = 2
25982024-04-03T05:21:11.645ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
25992024-04-03T05:21:11.645ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-04-03 05:12:15.995755 UTC dns_group = external generation = 2
26002024-04-03T05:21:11.645ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 23.487863ms iteration = 11
26012024-04-03T05:21:11.646ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:974 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
26022024-04-03T05:21:11.646ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 1c2282ea-206d-4df3-bfca-a35a264cd464 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
26032024-04-03T05:21:11.646ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
26042024-04-03T05:21:11.646ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-04-03 05:12:15.091055 UTC dns_group = internal generation = 1
26052024-04-03T05:21:11.646ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 24.08218ms iteration = 11
26062024-04-03T05:21:11.647ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 11 reason = Timeout
26072024-04-03T05:21:11.647ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("9f24d5b2-898d-40ee-a459-c6576fb9d418.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c33ab12a-21de-454f-9b76-9a78bfc5c720.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c33ab12a-21de-454f-9b76-9a78bfc5c720.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3664, tv_nsec: 326282712 } })
26082024-04-03T05:21:11.648ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
26092024-04-03T05:21:11.648ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.47857ms iteration = 11
26102024-04-03T05:21:11.675ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 20 reason = Timeout
26112024-04-03T05:21:11.675ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
26122024-04-03T05:21:11.675ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 20 reason = Timeout
26132024-04-03T05:21:11.675ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26142024-04-03T05:21:11.675ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 20 reason = Timeout
26152024-04-03T05:21:11.675ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
26162024-04-03T05:21:11.675ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 20 reason = Timeout
26172024-04-03T05:21:11.675ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
26182024-04-03T05:21:11.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26192024-04-03T05:21:11.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
26202024-04-03T05:21:11.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
26212024-04-03T05:21:11.676ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:982 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 zone_address = fd00:1122:3344:101::2
26222024-04-03T05:21:11.676ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
26232024-04-03T05:21:11.676ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26242024-04-03T05:21:11.676ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "39da69b0-74bd-4630-a2bc-7842ac29d27a", "content-length": "26", "date": "Wed, 03 Apr 2024 05:21:11 GMT"} })
26252024-04-03T05:21:11.677ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:987 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
26262024-04-03T05:21:11.677ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3664, tv_nsec: 355946127 } })
26272024-04-03T05:21:11.677ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-54b5d8ce-2f89-4588-80f9-2a71dfa53de1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3664, tv_nsec: 356102799 } })
26282024-04-03T05:21:11.677ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
26292024-04-03T05:21:11.677ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.531223ms iteration = 20
26302024-04-03T05:21:11.678ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:974
26312024-04-03T05:21:11.678ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
26322024-04-03T05:21:11.678ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:974
26332024-04-03T05:21:11.679ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
26342024-04-03T05:21:11.679ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26352024-04-03T05:21:11.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26362024-04-03T05:21:11.680ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26372024-04-03T05:21:11.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26382024-04-03T05:21:11.694ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26392024-04-03T05:21:11.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26402024-04-03T05:21:11.695ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26412024-04-03T05:21:11.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26422024-04-03T05:21:11.696ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26432024-04-03T05:21:11.699ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 54b5d8ce-2f89-4588-80f9-2a71dfa53de1, lookup_type: ById(54b5d8ce-2f89-4588-80f9-2a71dfa53de1) } result = Ok(())
26442024-04-03T05:21:11.699ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26452024-04-03T05:21:11.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26462024-04-03T05:21:11.700ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 25.418459ms iteration = 20
26472024-04-03T05:21:11.784ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
26482024-04-03T05:21:11.784ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
26492024-04-03T05:21:11.785ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "625119b5-98a7-4e53-ae0f-638c1c007cfa", "content-length": "26", "date": "Wed, 03 Apr 2024 05:21:11 GMT"} })
26502024-04-03T05:21:11.785ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
26512024-04-03T05:21:11.787ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
26522024-04-03T05:21:11.882ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 207.405343ms iteration = 20
26532024-04-03T05:21:11.884ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:982 zone_address = fd00:1122:3344:101::2
26542024-04-03T05:21:11.884ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
26552024-04-03T05:21:11.884ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "5a58d223-b693-48bc-8ce5-46f820a8221c", "content-length": "26", "date": "Wed, 03 Apr 2024 05:21:11 GMT"} })
26562024-04-03T05:21:11.884ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:987 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
26572024-04-03T05:21:11.885ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1019 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
26582024-04-03T05:21:11.945ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
26592024-04-03T05:21:11.946ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "87fe268c-6ae3-499f-8d1c-752c6d10c473", "content-length": "2", "date": "Wed, 03 Apr 2024 05:21:11 GMT"} })
26602024-04-03T05:21:11.947ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
26612024-04-03T05:21:11.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
26622024-04-03T05:21:11.949ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 274.654489ms iteration = 20
26632024-04-03T05:21:11.952ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 13 reason = Timeout
26642024-04-03T05:21:11.995ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26652024-04-03T05:21:11.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26662024-04-03T05:21:11.997ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:141 port = SwitchPort { id: 9abb0c53-3a74-441c-8a11-a2b0939760e0, rack_id: 38b3f10b-8962-43bd-8d32-8172eeecefb9, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(0a16a58f-d519-4d96-87f3-b4294bb539f8) } rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
26672024-04-03T05:21:11.997ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26682024-04-03T05:21:11.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26692024-04-03T05:21:12.004ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
26702024-04-03T05:21:12.004ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
26712024-04-03T05:21:12.005ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "70a7b438-0c00-4e04-a6bc-b308c778e8c7", "content-length": "121", "date": "Wed, 03 Apr 2024 05:21:11 GMT"} })
26722024-04-03T05:21:12.005ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1418 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
26732024-04-03T05:21:12.005ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1445 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 0a16a58f-d519-4d96-87f3-b4294bb539f8
26742024-04-03T05:21:12.005ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:364 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
26752024-04-03T05:21:12.005ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:368 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 })}}
26762024-04-03T05:21:12.005ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:377 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
26772024-04-03T05:21:12.005ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:383 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 routes = {}
26782024-04-03T05:21:12.005ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:403 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9
26792024-04-03T05:21:12.005ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26802024-04-03T05:21:12.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26812024-04-03T05:21:12.006ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
26822024-04-03T05:21:12.006ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9302f6dd-e2ca-4f97-b6ce-9a0dd8160668", "content-length": "2", "date": "Wed, 03 Apr 2024 05:21:11 GMT"} })
26832024-04-03T05:21:12.006ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
26842024-04-03T05:21:12.006ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "60c1b91a-39ce-4634-a770-60e136bb2443", "content-length": "52", "date": "Wed, 03 Apr 2024 05:21:11 GMT"} })
26852024-04-03T05:21:12.006ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:452 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch_location = Switch0
26862024-04-03T05:21:12.006ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
26872024-04-03T05:21:12.035ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
26882024-04-03T05:21:12.035ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "d2871854-bafc-44a6-9d46-3bc2dd4a4431", "date": "Wed, 03 Apr 2024 05:21:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
26892024-04-03T05:21:12.035ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
26902024-04-03T05:21:12.036ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "920c2ca5-15e6-4734-89e2-ba2374300c6e", "date": "Wed, 03 Apr 2024 05:21:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
26912024-04-03T05:21:12.036ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
26922024-04-03T05:21:12.038ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
26932024-04-03T05:21:12.038ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 86.35945ms iteration = 13
26942024-04-03T05:21:12.056ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "4393e1e2-4d1b-4c44-8b02-b2c3c196ec1c", "date": "Wed, 03 Apr 2024 05:21:12 GMT"} })
26952024-04-03T05:21:12.056ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:617 rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 switch = Switch0
26962024-04-03T05:21:12.056ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
26972024-04-03T05:21:12.056ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 38b3f10b-8962-43bd-8d32-8172eeecefb9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9c3d95f7-d60c-4f28-ae7f-c372dcb58bb1", "content-length": "452", "date": "Wed, 03 Apr 2024 05:21:12 GMT"} })
26982024-04-03T05:21:12.056ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26992024-04-03T05:21:12.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27002024-04-03T05:21:12.059ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27012024-04-03T05:21:12.060ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27022024-04-03T05:21:12.062ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27032024-04-03T05:21:12.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27042024-04-03T05:21:12.063ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27052024-04-03T05:21:12.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27062024-04-03T05:21:12.070ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 431.723092ms iteration = 20
27072024-04-03T05:21:12.296ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 13 reason = Timeout
27082024-04-03T05:21:12.326ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
27092024-04-03T05:21:12.356ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
27102024-04-03T05:21:12.386ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
27112024-04-03T05:21:12.386ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 156 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45789 req_id = 470e8542-a3aa-40e7-8d03-7a906f8a0d9c response_code = 200 uri = /metrics/collect/4b24c6b8-5b39-46e6-b7f8-38305b0660eb
27122024-04-03T05:21:12.387ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "21aa747c-aee1-459f-81e8-2dbaa8c5d1fd", "date": "Wed, 03 Apr 2024 05:21:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
27132024-04-03T05:21:12.387ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
27142024-04-03T05:21:12.388ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "d70b08ef-6721-46f1-beba-1fc73f4ac2f4", "date": "Wed, 03 Apr 2024 05:21:12 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
27152024-04-03T05:21:12.388ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
27162024-04-03T05:21:12.390ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "129f50df-1008-43ac-bf8a-e275d356b339", "date": "Wed, 03 Apr 2024 05:21:11 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
27172024-04-03T05:21:12.390ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
27182024-04-03T05:21:12.392ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
27192024-04-03T05:21:12.392ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 96.435205ms iteration = 13