Max level shown:
1 [ Dec 24 02:29:39 Disabled. ]
2 [ Dec 24 02:29:39 Rereading configuration. ]
3 [ Dec 24 02:29:40 Enabled. ]
4 [ Dec 24 02:29:40 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
5 [ Dec 24 02:29:40 Method "start" exited with status 0. ]
6 [ Dec 24 02:29:40 Rereading configuration. ]
7 [ Dec 24 02:29:40 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 214"). ]
8 + CONTRACT=''
9 + getopts c: o
10 + CONTRACT=214
11 + getopts c: o
12 + [[ -n 214 ]]
13 + pgrep -c 214
14 + PARENT_PID=2284
15 + [[ -n 2284 ]]
16 + pgrep -P 2284
17 + CHILD_PID=2285
18 + [[ -n 2285 ]]
19 + kill -USR1 2285
20 [ Dec 24 02:29:40 Method "refresh" exited with status 0. ]
21 [ Dec 24 02:29:40 Stopping because all processes in service exited. ]
22 [ Dec 24 02:29:40 Executing stop method (:kill). ]
23 [ Dec 24 02:29:40 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
24 [ Dec 24 02:29:40 Method "start" exited with status 0. ]
252025-12-24T02:29:40.742ZINFOslog-rs (mgd): signal handler waiting for context module = admin unit = signal
262025-12-24T02:29:40.742ZINFOslog-rs (bgp): dispatcher started listen_address = [::]:179 module = neighbor unit = dispatcher
272025-12-24T02:29:40.742ZDEBGslog-rs (bgp): listener bind: [::]:179 listen_address = [::]:179 module = neighbor unit = dispatcher
282025-12-24T02:29:40.764ZINFOslog-rs (mgd): signal handler got context module = admin unit = signal
292025-12-24T02:29:40.764ZINFOslog-rs (mgd): starting bgp routers: {} module = admin unit = daemon
302025-12-24T02:29:40.764ZINFOslog-rs (mgd): starting bfd sessions: [] module = admin unit = daemon
312025-12-24T02:29:40.765ZERROslog-rs (mgd): failed to start stats server: underlay address not found error = underlay address not found module = admin params = hostname oxz_switch, rack f320ad23-ac5f-4346-bdea-cf23700e7429, sled e6530454-30db-4c69-980a-1be890c6113f unit = daemon
322025-12-24T02:29:40.767ZINFOslog-rs: listening on [::]:4676
332025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bestpath/config/fanout unit = api_server versions = all ending with 2.0.0
342025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bestpath/config/fanout unit = api_server versions = all ending with 2.0.0
352025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bfd/peers unit = api_server versions = all
362025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bfd/peers unit = api_server versions = all
372025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bfd/peers/{addr} unit = api_server versions = all
382025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/clear/neighbor unit = api_server versions = all
392025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/checker unit = api_server versions = all
402025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/checker unit = api_server versions = all
412025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/checker unit = api_server versions = all
422025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/checker unit = api_server versions = all
432025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/neighbor unit = api_server versions = all
442025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/neighbor unit = api_server versions = all
452025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/neighbor unit = api_server versions = all
462025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/neighbor unit = api_server versions = all
472025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/neighbors unit = api_server versions = all
482025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/origin4 unit = api_server versions = all
492025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/origin4 unit = api_server versions = all
502025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/origin4 unit = api_server versions = all
512025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/origin4 unit = api_server versions = all
522025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
532025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
542025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
552025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
562025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/router unit = api_server versions = all
572025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/router unit = api_server versions = all
582025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/router unit = api_server versions = all
592025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/router unit = api_server versions = all
602025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/routers unit = api_server versions = all
612025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/shaper unit = api_server versions = all
622025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/shaper unit = api_server versions = all
632025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/shaper unit = api_server versions = all
642025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/shaper unit = api_server versions = all
652025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/history/fsm unit = api_server versions = all starting from 2.0.0
662025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/history/message unit = api_server versions = all starting from 2.0.0
672025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/message-history unit = api_server versions = all ending with 2.0.0
682025-12-24T02:29:40.767ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/omicron/apply unit = api_server versions = all
692025-12-24T02:29:40.768ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/exported unit = api_server versions = all
702025-12-24T02:29:40.768ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/imported unit = api_server versions = all ending with 2.0.0
712025-12-24T02:29:40.768ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/neighbors unit = api_server versions = all ending with 2.0.0
722025-12-24T02:29:40.768ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/neighbors unit = api_server versions = all starting from 2.0.0
732025-12-24T02:29:40.768ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/selected unit = api_server versions = all ending with 2.0.0
742025-12-24T02:29:40.768ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/config/bestpath/fanout unit = api_server versions = all starting from 2.0.0
752025-12-24T02:29:40.768ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /rib/config/bestpath/fanout unit = api_server versions = all starting from 2.0.0
762025-12-24T02:29:40.768ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/status/imported unit = api_server versions = all starting from 2.0.0
772025-12-24T02:29:40.768ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/status/selected unit = api_server versions = all starting from 2.0.0
782025-12-24T02:29:40.768ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /static/route4 unit = api_server versions = all
792025-12-24T02:29:40.768ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /static/route4 unit = api_server versions = all
802025-12-24T02:29:40.768ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /static/route4 unit = api_server versions = all
812025-12-24T02:29:40.768ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /static/route6 unit = api_server versions = all starting from 2.0.0
822025-12-24T02:29:40.768ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /static/route6 unit = api_server versions = all starting from 2.0.0
832025-12-24T02:29:40.768ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /static/route6 unit = api_server versions = all starting from 2.0.0
842025-12-24T02:29:40.768ZINFOslog-rs (mgd): listening local_addr = [::]:4676 module = admin unit = api_server
852025-12-24T02:29:40.768ZWARNslog-rs (mg-lower): failed to ensure TEP address fd1b:bf08:61dd:7c72::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd1b:bf08:61dd:7c72::1 unit = dpd
862025-12-24T02:29:40.768ZDEBGslog-rs (mgd): successfully registered DTrace USDT probes local_addr = [::]:4676 module = admin unit = api_server
872025-12-24T02:29:40.768ZINFOslog-rs: admin: server started
882025-12-24T02:29:41.768ZWARNslog-rs (mg-lower): failed to ensure TEP address fd1b:bf08:61dd:7c72::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd1b:bf08:61dd:7c72::1 unit = dpd
892025-12-24T02:29:42.768ZWARNslog-rs (mg-lower): failed to ensure TEP address fd1b:bf08:61dd:7c72::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd1b:bf08:61dd:7c72::1 unit = dpd
902025-12-24T02:29:43.768ZWARNslog-rs (mg-lower): failed to ensure TEP address fd1b:bf08:61dd:7c72::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd1b:bf08:61dd:7c72::1 unit = dpd
912025-12-24T02:29:44.769ZWARNslog-rs (mg-lower): failed to ensure TEP address fd1b:bf08:61dd:7c72::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd1b:bf08:61dd:7c72::1 unit = dpd
922025-12-24T02:29:45.769ZWARNslog-rs (mg-lower): failed to ensure TEP address fd1b:bf08:61dd:7c72::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd1b:bf08:61dd:7c72::1 unit = dpd
932025-12-24T02:29:46.812ZWARNslog-rs (mg-lower): failed to ensure TEP address fd1b:bf08:61dd:7c72::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd1b:bf08:61dd:7c72::1 unit = dpd
942025-12-24T02:29:47.811ZWARNslog-rs (mg-lower): failed to ensure TEP address fd1b:bf08:61dd:7c72::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd1b:bf08:61dd:7c72::1 unit = dpd
952025-12-24T02:29:48.811ZWARNslog-rs (mg-lower): failed to ensure TEP address fd1b:bf08:61dd:7c72::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd1b:bf08:61dd:7c72::1 unit = dpd
962025-12-24T02:29:49.812ZWARNslog-rs (mg-lower): failed to ensure TEP address fd1b:bf08:61dd:7c72::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd1b:bf08:61dd:7c72::1 unit = dpd
972025-12-24T02:29:50.812ZWARNslog-rs (mg-lower): failed to ensure TEP address fd1b:bf08:61dd:7c72::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd1b:bf08:61dd:7c72::1 unit = dpd
98 [ Dec 24 02:29:52 Rereading configuration. ]
99 [ Dec 24 02:29:52 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 219"). ]
100 + CONTRACT=''
101 + getopts c: o
102 + CONTRACT=219
103 + getopts c: o
104 + [[ -n 219 ]]
105 + pgrep -c 219
106 + PARENT_PID=2311
107 + [[ -n 2311 ]]
108 + pgrep -P 2311
109 + CHILD_PID=2313
110 + [[ -n 2313 ]]
111 + kill -USR1 2313
1122025-12-24T02:29:52.059ZINFOslog-rs (mgd): handling smf refresh module = admin unit = smf
113 [ Dec 24 02:29:52 Method "refresh" exited with status 0. ]
1142025-12-24T02:29:52.061ZINFOslog-rs (mgd): starting stats server on smf refresh module = admin unit = smf
1152025-12-24T02:29:52.061ZINFOslog-rs (mgd): started stats server on smf refresh module = admin unit = smf
116 Dec 24 02:29:52.062 DEBG registered DTrace probes
117 Dec 24 02:29:52.062 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
118 Dec 24 02:29:52.062 INFO listening, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.5/src/server.rs:328
119 Dec 24 02:29:52.062 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
120 Dec 24 02:29:52.062 DEBG Nexus IP not provided, will use DNS to resolve it
121 Dec 24 02:29:52.062 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/b36f032/internal-dns/resolver/src/resolver.rs:144
122 Dec 24 02:29:52.063 DEBG starting producer registration task
123 Dec 24 02:29:52.063 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:4677, producer_id: 2c6ba0f7-3bc6-4f92-9b27-312ac346d8e3, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/b36f032/oximeter/producer/src/lib.rs:279
124 Dec 24 02:29:52.063 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
1252025-12-24T02:29:52.386ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::1]:39301 unit = api_server
1262025-12-24T02:29:52.386ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = PUT module = admin remote_addr = [fd00:1122:3344:101::1]:39301 req_id = 716f5739-f5f2-4a0d-b0a8-83cc0b083dd0 unit = api_server uri = /static/route4 version = 2.0.0
1272025-12-24T02:29:52.387ZINFOslog-rs (mgd): request completed latency_us = 588 local_addr = [::]:4676 method = PUT module = admin remote_addr = [fd00:1122:3344:101::1]:39301 req_id = 716f5739-f5f2-4a0d-b0a8-83cc0b083dd0 response_code = 204 unit = api_server uri = /static/route4
1282025-12-24T02:29:52.388ZERROslog-rs (mg-lower): handling change failed: libnet error route io error No such process (os error 3) error = libnet error route io error No such process (os error 3) module = sync unit = event_loop
1292025-12-24T02:29:52.388ZINFOslog-rs (mg-lower): restarting sync loop module = sync unit = event_loop
130 Dec 24 02:29:57.065 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5616f378-ae97-4dbc-8124-55a87ab6e6f9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c3303866-f250-4713-8e04-c8b0328918d6.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("fae05261-2cc7-4def-871e-fd71a930a30e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("fae05261-2cc7-4def-871e-fd71a930a30e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 655, tv_nsec: 690226614 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
131 Dec 24 02:29:57.066 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
132 Dec 24 02:29:57.066 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task