| 1 | | [ Jul 23 10:11:07 Disabled. ] |
| 2 | | [ Jul 23 10:11:07 Rereading configuration. ] |
| 3 | | [ Jul 23 10:11:08 Enabled. ] |
| 4 | | [ Jul 23 10:11:08 Rereading configuration. ] |
| 5 | | [ Jul 23 10:11:08 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ] |
| 6 | | [ Jul 23 10:11:08 Method "start" exited with status 0. ] |
| 7 | 2025-07-23T10:11:08.408Z | INFO | slog-rs: signal handler waiting for context
|
| 8 | 2025-07-23T10:11:08.429Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 9 | 2025-07-23T10:11:08.429Z | ERRO | slog-rs: failed to start stats server: underlay address not found
|
| 10 | 2025-07-23T10:11:08.429Z | INFO | slog-rs: signal handler waiting got context
|
| 11 | 2025-07-23T10:11:08.430Z | INFO | slog-rs: admin: listening on [::1]:8000
|
| 12 | 2025-07-23T10:11:09.430Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 13 | 2025-07-23T10:11:10.429Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 14 | 2025-07-23T10:11:11.429Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 15 | 2025-07-23T10:11:12.430Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 16 | 2025-07-23T10:11:13.430Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 17 | 2025-07-23T10:11:14.430Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 18 | 2025-07-23T10:11:15.430Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 19 | 2025-07-23T10:11:16.430Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 20 | 2025-07-23T10:11:17.430Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 21 | 2025-07-23T10:11:18.430Z | WARN | slog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
|
| 22 | 2025-07-23T10:11:19.430Z | INFO | slog-rs: [tfportrear0_0] sm initialized with addr fe80::38f8:a2ff:fe92:538f on if tfportrear0_0 index 3
|
| 23 | 2025-07-23T10:11:19.430Z | DEBG | slog-rs: [tfportrear0_0] starting discovery handler
|
| 24 | 2025-07-23T10:11:19.431Z | INFO | slog-rs: [tfportrear0_0] nbr is fe80::f882:2bff:feaf:fc9e@buskin server
|
| 25 | 2025-07-23T10:11:19.431Z | DEBG | slog-rs: [tfportrear0_0] transition solicit -> exchange
|
| 26 | 2025-07-23T10:11:19.431Z | INFO | slog-rs: [tfportrear0_0] exchange: listening on [fe80::38f8:a2ff:fe92:538f]:56797
|
| 27 | 2025-07-23T10:11:19.431Z | INFO | slog-rs: [tfportrear0_0] waiting for exchange server to start
|
| 28 | | [ Jul 23 10:11:19 Rereading configuration. ] |
| 29 | | [ Jul 23 10:11:19 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 216"). ] |
| 30 | | + CONTRACT='' |
| 31 | | + getopts c: o |
| 32 | | + CONTRACT=216 |
| 33 | | + getopts c: o |
| 34 | | + [[ -n 216 ]] |
| 35 | | + pgrep -c 216 |
| 36 | | + PARENT_PID=2286 |
| 37 | | + [[ -n 2286 ]] |
| 38 | | + pgrep -P 2286 |
| 39 | | + CHILD_PID=2289 |
| 40 | | + [[ -n 2289 ]] |
| 41 | | + kill -USR1 2289 |
| 42 | 2025-07-23T10:11:19.797Z | INFO | slog-rs: handling smf refresh
|
| 43 | | [ Jul 23 10:11:19 Method "refresh" exited with status 0. ] |
| 44 | 2025-07-23T10:11:19.801Z | INFO | slog-rs: starting stats server on smf refresh
|
| 45 | | Jul 23 10:11:19.802 DEBG registered DTrace probes |
| 46 | | Jul 23 10:11:19.802 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot |
| 47 | | Jul 23 10:11:19.802 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:328 |
| 48 | | Jul 23 10:11:19.803 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot |
| 49 | | Jul 23 10:11:19.803 DEBG Nexus IP not provided, will use DNS to resolve it |
| 50 | | Jul 23 10:11:19.803 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/internal-dns/resolver/src/resolver.rs:111 |
| 51 | | Jul 23 10:11:19.803 DEBG starting producer registration task |
| 52 | | Jul 23 10:11:19.803 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: a2826c11-418c-4af4-a55c-46138ee2938d, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:279 |
| 53 | | Jul 23 10:11:19.803 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task |
| 54 | 2025-07-23T10:11:21.016Z | INFO | slog-rs: advertise tunnel: {
TunnelOrigin {
overlay_prefix: V4(
Ipv4Net {
addr: 0.0.0.0,
width: 0,
},
),
boundary_addr: fd88:4092:f937:3b05::1,
vni: 99,
metric: 18446744073709551615,
},
}
|
| 55 | 2025-07-23T10:11:21.433Z | WARN | slog-rs: [tfportrear0_0] exchange pull: hyper client error: client error (Connect)
|
| 56 | 2025-07-23T10:11:21.434Z | INFO | slog-rs: [tfportrear0_0] sending 5 routes to dendrite
|
| 57 | 2025-07-23T10:11:21.434Z | DEBG | slog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
|
| 58 | 2025-07-23T10:11:21.435Z | INFO | slog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::f882:2bff:feaf:fc9e on port Rear(Rear("rear0"))/0
|
| 59 | 2025-07-23T10:11:21.436Z | INFO | slog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::f882:2bff:feaf:fc9e on port Rear(Rear("rear0"))/0
|
| 60 | 2025-07-23T10:11:21.437Z | INFO | slog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::f882:2bff:feaf:fc9e on port Rear(Rear("rear0"))/0
|
| 61 | 2025-07-23T10:11:21.437Z | INFO | slog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::f882:2bff:feaf:fc9e on port Rear(Rear("rear0"))/0
|
| 62 | 2025-07-23T10:11:21.437Z | INFO | slog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::f882:2bff:feaf:fc9e on port Rear(Rear("rear0"))/0
|
| 63 | 2025-07-23T10:11:21.437Z | INFO | slog-rs: [tfportrear0_0] removing routes 0 from dendrite
|
| 64 | 2025-07-23T10:11:21.437Z | DEBG | slog-rs: [tfportrear0_0] redistributing update to 0 peers
|
| 65 | | Jul 23 10:11:24.805 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("503f98a0-54fd-4258-b2d8-a761939d1032.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d8901a1f-81d9-4e2e-a334-fe58271d3891.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 632, tv_nsec: 161598421 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 66 | | Jul 23 10:11:24.806 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 67 | | Jul 23 10:11:24.806 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 68 | | Jul 23 10:11:39.105 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task |
| 69 | | Jul 23 10:11:39.105 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 20.004910452s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371 |
| 70 | | Jul 23 10:11:39.420 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("503f98a0-54fd-4258-b2d8-a761939d1032.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d8901a1f-81d9-4e2e-a334-fe58271d3891.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 647, tv_nsec: 478352827 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 71 | | Jul 23 10:11:39.421 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 72 | | Jul 23 10:11:39.421 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 73 | | Jul 23 10:11:54.463 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task |
| 74 | | Jul 23 10:11:54.463 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 35.320644726s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371 |
| 75 | | Jul 23 10:11:54.790 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("503f98a0-54fd-4258-b2d8-a761939d1032.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d8901a1f-81d9-4e2e-a334-fe58271d3891.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 662, tv_nsec: 806907925 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 76 | | Jul 23 10:11:54.791 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 77 | | Jul 23 10:11:54.791 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 78 | | Jul 23 10:12:09.834 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task |
| 79 | | Jul 23 10:12:09.834 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 50.649871048s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371 |
| 80 | | Jul 23 10:12:10.802 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("503f98a0-54fd-4258-b2d8-a761939d1032.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d8901a1f-81d9-4e2e-a334-fe58271d3891.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 678, tv_nsec: 774774504 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 81 | | Jul 23 10:12:10.802 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 82 | | Jul 23 10:12:10.802 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 83 | | Jul 23 10:12:25.845 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task |
| 84 | | Jul 23 10:12:25.845 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 66.617802449s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371 |
| 85 | | Jul 23 10:12:27.461 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("503f98a0-54fd-4258-b2d8-a761939d1032.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d8901a1f-81d9-4e2e-a334-fe58271d3891.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 695, tv_nsec: 388916446 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 86 | | Jul 23 10:12:27.461 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 87 | | Jul 23 10:12:27.462 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 88 | | Jul 23 10:12:42.503 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task |
| 89 | | Jul 23 10:12:42.504 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 83.231154875s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371 |
| 90 | | Jul 23 10:12:44.688 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("503f98a0-54fd-4258-b2d8-a761939d1032.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d8901a1f-81d9-4e2e-a334-fe58271d3891.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 712, tv_nsec: 569469145 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 91 | | Jul 23 10:12:44.689 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 92 | | Jul 23 10:12:44.689 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 93 | 2025-07-23T10:12:48.930Z | WARN | slog-rs: [tfportrear0_0] neighbor buskin@fe80::f882:2bff:feaf:fc9e missed solicit interval
|
| 94 | | Jul 23 10:12:59.708 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task |
| 95 | | Jul 23 10:12:59.708 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 100.411855788s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371 |
| 96 | | Jul 23 10:13:09.891 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("503f98a0-54fd-4258-b2d8-a761939d1032.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d8901a1f-81d9-4e2e-a334-fe58271d3891.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 737, tv_nsec: 755767507 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 97 | | Jul 23 10:13:09.892 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 98 | | Jul 23 10:13:09.892 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 99 | | Jul 23 10:13:24.897 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task |
| 100 | | Jul 23 10:13:24.897 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 125.59822004s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371 |
| 101 | 2025-07-23T10:13:36.423Z | WARN | slog-rs: [tfportrear0_0] neighbor buskin@fe80::f882:2bff:feaf:fc9e missed solicit interval
|
| 102 | | Jul 23 10:13:37.298 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("503f98a0-54fd-4258-b2d8-a761939d1032.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d8901a1f-81d9-4e2e-a334-fe58271d3891.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 765, tv_nsec: 144050277 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 103 | | Jul 23 10:13:37.309 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 104 | | Jul 23 10:13:37.405 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 105 | | Jul 23 10:13:52.417 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task |
| 106 | | Jul 23 10:13:52.420 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 153.120460338s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371 |
| 107 | | Jul 23 10:14:36.416 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("503f98a0-54fd-4258-b2d8-a761939d1032.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d8901a1f-81d9-4e2e-a334-fe58271d3891.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("ee826bb4-083c-4a45-85c5-33ab74f68864.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 824, tv_nsec: 272065277 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver |
| 108 | | Jul 23 10:14:36.417 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task |
| 109 | | Jul 23 10:14:36.419 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task |
| 110 | | Jul 23 10:14:36.431 DEBG client response, result: Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "05b4b44c-fa5c-4e0d-a689-8667c9696672", "content-length": "41", "date": "Wed, 23 Jul 2025 10:14:36 GMT"} }), component: producer-registration-task |
| 111 | | Jul 23 10:14:36.431 DEBG successfully registered with Nexus, pausing until time to renew lease, wait_period: 150s, lease_duration: 600s, component: producer-registration-task |
| 112 | 2025-07-23T10:14:42.675Z | INFO | slog-rs: withdraw tunnel: {
TunnelOrigin {
overlay_prefix: V4(
Ipv4Net {
addr: 0.0.0.0,
width: 0,
},
),
boundary_addr: fd88:4092:f937:3b05::1,
vni: 99,
metric: 18446744073709551615,
},
}
|
| 113 | 2025-07-23T10:14:42.683Z | INFO | slog-rs: advertise tunnel: {
TunnelOrigin {
overlay_prefix: V4(
Ipv4Net {
addr: 0.0.0.0,
width: 0,
},
),
boundary_addr: fd88:4092:f937:3b05::1,
vni: 99,
metric: 18446744073709551615,
},
}
|
| 114 | 2025-07-23T10:14:43.001Z | INFO | slog-rs: withdraw tunnel: {
TunnelOrigin {
overlay_prefix: V4(
Ipv4Net {
addr: 0.0.0.0,
width: 0,
},
),
boundary_addr: fd88:4092:f937:3b05::1,
vni: 99,
metric: 18446744073709551615,
},
}
|
| 115 | 2025-07-23T10:14:43.003Z | INFO | slog-rs: advertise tunnel: {
TunnelOrigin {
overlay_prefix: V4(
Ipv4Net {
addr: 0.0.0.0,
width: 0,
},
),
boundary_addr: fd88:4092:f937:3b05::1,
vni: 99,
metric: 18446744073709551615,
},
}
|
| 116 | 2025-07-23T10:14:43.255Z | INFO | slog-rs: withdraw tunnel: {
TunnelOrigin {
overlay_prefix: V4(
Ipv4Net {
addr: 0.0.0.0,
width: 0,
},
),
boundary_addr: fd88:4092:f937:3b05::1,
vni: 99,
metric: 18446744073709551615,
},
}
|
| 117 | 2025-07-23T10:14:43.257Z | INFO | slog-rs: advertise tunnel: {
TunnelOrigin {
overlay_prefix: V4(
Ipv4Net {
addr: 0.0.0.0,
width: 0,
},
),
boundary_addr: fd88:4092:f937:3b05::1,
vni: 99,
metric: 18446744073709551615,
},
}
|
| 118 | | Jul 23 10:14:51.218 INFO accepted connection, remote_addr: [fd00:1122:3344:101::d]:42247, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 |
| 119 | | Jul 23 10:14:51.226 INFO request completed, latency_us: 8416, response_code: 200, uri: /a2826c11-418c-4af4-a55c-46138ee2938d, method: GET, req_id: 9e625640-aec6-4d9a-963a-d84daf26d7b4, remote_addr: [fd00:1122:3344:101::d]:42247, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 |
| 120 | | Jul 23 10:14:52.219 INFO request completed, latency_us: 650, response_code: 200, uri: /a2826c11-418c-4af4-a55c-46138ee2938d, method: GET, req_id: 56f1bf23-f7ae-4204-9946-4656eb963d5f, remote_addr: [fd00:1122:3344:101::d]:42247, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 |
| 121 | | Jul 23 10:14:53.219 INFO request completed, latency_us: 733, response_code: 200, uri: /a2826c11-418c-4af4-a55c-46138ee2938d, method: GET, req_id: 0f1eb607-8fb7-433b-abbe-be1593b7425e, remote_addr: [fd00:1122:3344:101::d]:42247, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 |
| 122 | | Jul 23 10:14:54.218 INFO request completed, latency_us: 672, response_code: 200, uri: /a2826c11-418c-4af4-a55c-46138ee2938d, method: GET, req_id: 1e21f71a-c7ed-40a2-93bb-44f3c4bb075f, remote_addr: [fd00:1122:3344:101::d]:42247, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 |
| 123 | | Jul 23 10:14:55.219 INFO request completed, latency_us: 716, response_code: 200, uri: /a2826c11-418c-4af4-a55c-46138ee2938d, method: GET, req_id: 5dada532-c5ae-49a5-9e83-1504abda80e5, remote_addr: [fd00:1122:3344:101::d]:42247, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 |
| 124 | | Jul 23 10:14:56.219 INFO request completed, latency_us: 975, response_code: 200, uri: /a2826c11-418c-4af4-a55c-46138ee2938d, method: GET, req_id: 6cfac77a-c18c-4bab-aefc-9a2f37d6c8d0, remote_addr: [fd00:1122:3344:101::d]:42247, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 |
| 125 | | Jul 23 10:14:57.219 INFO request completed, latency_us: 714, response_code: 200, uri: /a2826c11-418c-4af4-a55c-46138ee2938d, method: GET, req_id: 8cc25aea-e150-475f-98e4-738a8e5f69a1, remote_addr: [fd00:1122:3344:101::d]:42247, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 |
| 126 | | Jul 23 10:14:58.219 INFO request completed, latency_us: 657, response_code: 200, uri: /a2826c11-418c-4af4-a55c-46138ee2938d, method: GET, req_id: 76323b19-f2ea-4372-b5d8-d7a4387a4ddc, remote_addr: [fd00:1122:3344:101::d]:42247, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 |
| 127 | | Jul 23 10:14:59.219 INFO request completed, latency_us: 608, response_code: 200, uri: /a2826c11-418c-4af4-a55c-46138ee2938d, method: GET, req_id: 70f355d1-d002-4e89-b33c-70aa2494bee5, remote_addr: [fd00:1122:3344:101::d]:42247, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 |
| 128 | | Jul 23 10:15:00.219 INFO request completed, latency_us: 614, response_code: 200, uri: /a2826c11-418c-4af4-a55c-46138ee2938d, method: GET, req_id: 80325335-ed67-4461-85b0-dd9cf2ce8c73, remote_addr: [fd00:1122:3344:101::d]:42247, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 |