Max level shown:
1 [ Feb 23 16:27:58 Disabled. ]
2 [ Feb 23 16:27:58 Rereading configuration. ]
3 [ Feb 23 16:27:59 Enabled. ]
4 [ Feb 23 16:27:59 Rereading configuration. ]
5 [ Feb 23 16:27:59 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
6 [ Feb 23 16:27:59 Method "start" exited with status 0. ]
72026-02-23T16:27:59.287ZINFOslog-rs: signal handler waiting for context
82026-02-23T16:27:59.311ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
92026-02-23T16:27:59.311ZERROslog-rs: failed to start stats server: underlay address not found
102026-02-23T16:27:59.311ZINFOslog-rs: signal handler waiting got context
112026-02-23T16:27:59.312ZINFOslog-rs: admin: listening on [::1]:8000
122026-02-23T16:27:59.312ZINFOslog-rs: admin: server started
132026-02-23T16:28:00.311ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
142026-02-23T16:28:01.311ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
152026-02-23T16:28:02.311ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
162026-02-23T16:28:03.311ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
172026-02-23T16:28:04.311ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
182026-02-23T16:28:05.311ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
192026-02-23T16:28:06.312ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
202026-02-23T16:28:07.312ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
212026-02-23T16:28:08.312ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
222026-02-23T16:28:09.312ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
232026-02-23T16:28:10.312ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::e88f:b4ff:fe4a:7149 on if tfportrear0_0 index 3
242026-02-23T16:28:10.312ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
252026-02-23T16:28:10.312ZINFOslog-rs: [tfportrear0_0] nbr is fe80::bcaf:45ff:fe49:618f@buskin server
262026-02-23T16:28:10.313ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
272026-02-23T16:28:10.313ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::e88f:b4ff:fe4a:7149]:56797
282026-02-23T16:28:10.313ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
29 [ Feb 23 16:28:10 Rereading configuration. ]
30 [ Feb 23 16:28:10 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 213"). ]
31 + CONTRACT=''
32 + getopts c: o
33 + CONTRACT=213
34 + getopts c: o
35 + [[ -n 213 ]]
36 + pgrep -c 213
37 + PARENT_PID=2285
38 + [[ -n 2285 ]]
39 + pgrep -P 2285
40 + CHILD_PID=2286
41 + [[ -n 2286 ]]
42 + kill -USR1 2286
432026-02-23T16:28:10.807ZINFOslog-rs: handling smf refresh
44 [ Feb 23 16:28:10 Method "refresh" exited with status 0. ]
452026-02-23T16:28:10.809ZINFOslog-rs: starting stats server on smf refresh
46 Feb 23 16:28:10.810 DEBG registered DTrace probes
47 Feb 23 16:28:10.810 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
48 Feb 23 16:28:10.811 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:328
49 Feb 23 16:28:10.811 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
50 Feb 23 16:28:10.811 DEBG Nexus IP not provided, will use DNS to resolve it
51 Feb 23 16:28:10.811 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/internal-dns/resolver/src/resolver.rs:144
52 Feb 23 16:28:10.811 DEBG starting producer registration task
53 Feb 23 16:28:10.812 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: 036fd130-43ec-4f98-955d-41eec2d4bef1, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/oximeter/producer/src/lib.rs:279
54 Feb 23 16:28:10.812 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
552026-02-23T16:28:12.158ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fdff:ee6a:bd4:e6f0::1, vni: 99, metric: 18446744073709551615, }, }
562026-02-23T16:28:12.315ZWARNslog-rs: [tfportrear0_0] exchange pull: hyper client error: client error (Connect)
572026-02-23T16:28:12.316ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
582026-02-23T16:28:12.316ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
592026-02-23T16:28:12.317ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::bcaf:45ff:fe49:618f on port Rear(Rear("rear0"))/0
602026-02-23T16:28:12.319ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::bcaf:45ff:fe49:618f on port Rear(Rear("rear0"))/0
612026-02-23T16:28:12.319ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::bcaf:45ff:fe49:618f on port Rear(Rear("rear0"))/0
622026-02-23T16:28:12.319ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::bcaf:45ff:fe49:618f on port Rear(Rear("rear0"))/0
632026-02-23T16:28:12.319ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::bcaf:45ff:fe49:618f on port Rear(Rear("rear0"))/0
642026-02-23T16:28:12.320ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
652026-02-23T16:28:12.320ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
66 Feb 23 16:28:15.813 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("22821aba-0e42-4251-bba1-cfd3795ce2e0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("898d20d1-89b6-49fd-bdf8-f7f44c462267.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e3fd166e-83b4-4105-9fe2-3b5421d608da.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e3fd166e-83b4-4105-9fe2-3b5421d608da.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1025, tv_nsec: 768742375 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
67 Feb 23 16:28:15.814 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
68 Feb 23 16:28:15.814 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
692026-02-23T16:28:27.943ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::bcaf:45ff:fe49:618f missed solicit interval
702026-02-23T16:28:29.944ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::bcaf:45ff:fe49:618f missed solicit interval
71 Feb 23 16:28:30.433 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
72 Feb 23 16:28:30.434 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 20.002869592s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/oximeter/producer/src/lib.rs:371
73 Feb 23 16:28:30.619 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("22821aba-0e42-4251-bba1-cfd3795ce2e0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("898d20d1-89b6-49fd-bdf8-f7f44c462267.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e3fd166e-83b4-4105-9fe2-3b5421d608da.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e3fd166e-83b4-4105-9fe2-3b5421d608da.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1040, tv_nsec: 956440691 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
74 Feb 23 16:28:30.620 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
75 Feb 23 16:28:30.620 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
762026-02-23T16:28:31.944ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::bcaf:45ff:fe49:618f missed solicit interval
772026-02-23T16:28:33.944ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::bcaf:45ff:fe49:618f missed solicit interval
78 Feb 23 16:28:45.580 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
79 Feb 23 16:28:45.581 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 35.190804344s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/oximeter/producer/src/lib.rs:371
80 Feb 23 16:28:45.914 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("22821aba-0e42-4251-bba1-cfd3795ce2e0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("898d20d1-89b6-49fd-bdf8-f7f44c462267.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e3fd166e-83b4-4105-9fe2-3b5421d608da.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e3fd166e-83b4-4105-9fe2-3b5421d608da.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1056, tv_nsec: 291829756 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
81 Feb 23 16:28:45.914 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
82 Feb 23 16:28:45.914 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
832026-02-23T16:28:59.950ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::bcaf:45ff:fe49:618f missed solicit interval
84 Feb 23 16:29:00.879 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
85 Feb 23 16:29:00.903 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 50.53213106s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/oximeter/producer/src/lib.rs:371
86 Feb 23 16:29:01.600 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("22821aba-0e42-4251-bba1-cfd3795ce2e0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("898d20d1-89b6-49fd-bdf8-f7f44c462267.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e3fd166e-83b4-4105-9fe2-3b5421d608da.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e3fd166e-83b4-4105-9fe2-3b5421d608da.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1072, tv_nsec: 20493757 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
87 Feb 23 16:29:01.601 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
88 Feb 23 16:29:01.604 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
892026-02-23T16:29:13.654ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::bcaf:45ff:fe49:618f missed solicit interval
90 Feb 23 16:29:16.588 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
91 Feb 23 16:29:16.594 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 66.279698781s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/oximeter/producer/src/lib.rs:371
92 Feb 23 16:29:19.499 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("22821aba-0e42-4251-bba1-cfd3795ce2e0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("898d20d1-89b6-49fd-bdf8-f7f44c462267.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e3fd166e-83b4-4105-9fe2-3b5421d608da.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e3fd166e-83b4-4105-9fe2-3b5421d608da.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1089, tv_nsec: 947870503 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
93 Feb 23 16:29:19.500 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
94 Feb 23 16:29:19.502 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
95 Feb 23 16:29:34.511 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
96 Feb 23 16:29:34.511 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 84.186138045s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/oximeter/producer/src/lib.rs:371
97 Feb 23 16:29:39.479 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("22821aba-0e42-4251-bba1-cfd3795ce2e0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("898d20d1-89b6-49fd-bdf8-f7f44c462267.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e3fd166e-83b4-4105-9fe2-3b5421d608da.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e3fd166e-83b4-4105-9fe2-3b5421d608da.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1109, tv_nsec: 921940094 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
98 Feb 23 16:29:39.480 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
99 Feb 23 16:29:39.480 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
100 Feb 23 16:29:54.482 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
101 Feb 23 16:29:54.482 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 104.157439379s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/194594d/oximeter/producer/src/lib.rs:371