Max level shown:
1 [ May 13 20:42:34 Disabled. ]
2 [ May 13 20:42:34 Rereading configuration. ]
3 [ May 13 20:42:47 Rereading configuration. ]
4 [ May 13 20:42:47 Rereading configuration. ]
5 [ May 13 20:42:47 Enabled. ]
6 [ May 13 20:42:47 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
7 [ May 13 20:42:47 Method "start" exited with status 0. ]
82024-05-13T20:42:47.391ZINFOslog-rs: signal handler waiting for context
92024-05-13T20:42:47.410ZINFOslog-rs: signal handler waiting got context
102024-05-13T20:42:47.411ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::9083:42ff:fe08:a2bb on if tfportrear0_0 index 3
112024-05-13T20:42:47.411ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
122024-05-13T20:42:47.411ZINFOslog-rs: admin: listening on [::1]:8000
13 [ May 13 20:42:48 Rereading configuration. ]
14 [ May 13 20:42:48 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 461"). ]
15 + CONTRACT=''
16 + getopts c: o
17 + CONTRACT=461
18 + getopts c: o
19 + [[ -n 461 ]]
20 + pgrep -c 461
21 + PARENT_PID=4353
22 + [[ -n 4353 ]]
23 + pgrep -P 4353
24 + CHILD_PID=4354
25 + [[ -n 4354 ]]
26 + kill -USR1 4354
272024-05-13T20:42:48.758ZINFOslog-rs: handling smf refresh
28 [ May 13 20:42:48 Method "refresh" exited with status 0. ]
292024-05-13T20:42:48.759ZINFOslog-rs: starting stats server on smf refresh
302024-05-13T20:42:48.760ZINFOslog-rs: resolving nexus
312024-05-13T20:42:48.760ZINFOslog-rs: new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53]
322024-05-13T20:42:48.762ZDEBGslog-rs: lookup_socket_v6 srv dns_name = _nexus._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("70953454-853e-4a68-95ed-c1ea05b22a92.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("902991b0-67e8-4f1b-9e26-7701c4f674d2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e2ae3bfc-0ab2-4114-858f-277e440078e4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("e2ae3bfc-0ab2-4114-858f-277e440078e4.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 4580, tv_nsec: 867486459 } })
33 May 13 20:42:48.764 DEBG registered DTrace probes
34 May 13 20:42:48.764 DEBG registered endpoint, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
35 May 13 20:42:48.764 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/33e158b/dropshot/src/server.rs:195
36 May 13 20:42:48.764 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
37 May 13 20:42:48.764 DEBG Nexus IP provided explicitly, will use it, addr: [fd00:1122:3344:101::b]:12221
38 May 13 20:42:48.764 DEBG starting producer registration task
39 May 13 20:42:48.764 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: 2d621e93-f564-42a9-8acf-8dce48684c17, route: //2d621e93-f564-42a9-8acf-8dce48684c17, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:284
40 May 13 20:42:48.765 DEBG registering / renewing oximeter producer lease with Nexus, component: producer-registration-task
41 May 13 20:42:48.765 DEBG using nexus address for registration, addr: [fd00:1122:3344:101::b]:12221, component: producer-registration-task
42 May 13 20:42:48.803 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
432024-05-13T20:42:49.411ZINFOslog-rs: [tfportrear0_0] nbr is fe80::68b6:88ff:fe57:73f6@buskin server
442024-05-13T20:42:49.411ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
452024-05-13T20:42:49.411ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::9083:42ff:fe08:a2bb]:56797
462024-05-13T20:42:49.412ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
472024-05-13T20:42:49.414ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
482024-05-13T20:42:49.414ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
492024-05-13T20:42:49.446ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::68b6:88ff:fe57:73f6 on port Rear(RearPort(0))/0
502024-05-13T20:42:49.447ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::68b6:88ff:fe57:73f6 on port Rear(RearPort(0))/0
512024-05-13T20:42:49.447ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::68b6:88ff:fe57:73f6 on port Rear(RearPort(0))/0
522024-05-13T20:42:49.448ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::68b6:88ff:fe57:73f6 on port Rear(RearPort(0))/0
532024-05-13T20:42:49.448ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::68b6:88ff:fe57:73f6 on port Rear(RearPort(0))/0
542024-05-13T20:42:49.448ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
552024-05-13T20:42:49.477ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
562024-05-13T20:42:49.993ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Prefix { addr: 0.0.0.0, len: 0, }, ), boundary_addr: fdeb:776c:76c9:6e99::1, vni: 99, metric: 18446744073709551615, }, }
572024-05-13T20:42:49.997ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Prefix { addr: 0.0.0.0, len: 0, }, ), boundary_addr: fdeb:776c:76c9:6e99::1, vni: 99, metric: 18446744073709551615, }, }
58 May 13 20:43:03.804 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::b)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
59 May 13 20:43:03.805 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out", delay: 317.804419ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
60 May 13 20:43:04.154 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
61 May 13 20:43:19.065 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::b)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
62 May 13 20:43:19.066 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out", delay: 561.486747ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
63 May 13 20:43:19.658 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
64 May 13 20:43:34.660 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::b)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
65 May 13 20:43:34.661 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out", delay: 842.82298ms, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
66 May 13 20:43:35.535 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
67 May 13 20:43:50.536 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::b)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
68 May 13 20:43:50.537 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out", delay: 1.468168283s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
69 May 13 20:43:52.035 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
70 May 13 20:44:07.037 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::b)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
71 May 13 20:44:07.037 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out", delay: 2.897063245s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
72 May 13 20:44:09.965 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
732024-05-13T20:44:13.323ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
742024-05-13T20:44:15.324ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
752024-05-13T20:44:17.325ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
762024-05-13T20:44:19.326ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
772024-05-13T20:44:21.327ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
782024-05-13T20:44:23.328ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
79 May 13 20:44:24.929 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::b)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
80 May 13 20:44:24.930 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out", delay: 4.331614989s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
812024-05-13T20:44:25.329ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
822024-05-13T20:44:27.330ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
83 May 13 20:44:29.281 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
842024-05-13T20:44:29.331ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
852024-05-13T20:44:31.332ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
862024-05-13T20:44:33.333ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
872024-05-13T20:44:35.334ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
882024-05-13T20:44:37.335ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
892024-05-13T20:44:39.336ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
902024-05-13T20:44:41.337ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
912024-05-13T20:44:43.338ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval
92 May 13 20:44:44.241 DEBG client response, result: Err(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::b)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, source: TimedOut }), component: producer-registration-task
93 May 13 20:44:44.242 WARN failed to register as a producer with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out", delay: 17.927584856s, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-d039c41f152bda83/ee901b0/oximeter/producer/src/lib.rs:426
942024-05-13T20:44:45.338ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::68b6:88ff:fe57:73f6 missed solicit interval