Max level shown:
1 [ Sep 19 20:46:05 Enabled. ]
2 [ Sep 19 20:46:05 Rereading configuration. ]
3 [ Sep 19 20:46:09 Rereading configuration. ]
4 [ Sep 19 20:46:13 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible -l [fd00:1122:3344:101::12]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-09-19T20:46:14.027ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 779775d5130ff7a4836f52f48b7e64d1479ee104 Commit timestamp: 2025-08-09T00:27:59.000000000Z branch: main rustc: 1.86.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72025-09-19T20:46:14.030ZINFOcrucible-agent: dataset: "oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible"
82025-09-19T20:46:14.030ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::12]:32345
92025-09-19T20:46:14.030ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102025-09-19T20:46:14.047ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112025-09-19T20:46:14.223ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::12]:32345
122025-09-19T20:47:02.307ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::b]:60229
132025-09-19T20:47:02.358ZINFOcrucible-agent (datafile): region 62add0c2-3a4d-4471-bec1-2ae04c1e8e12 state: Requested
142025-09-19T20:47:02.365ZINFOcrucible-agent (dropshot): request completed latency_us = 29847 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:60229 req_id = b9d79208-8dcc-4698-a208-e430475027b4 response_code = 200 uri = /crucible/0/regions
152025-09-19T20:47:02.370ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
162025-09-19T20:47:02.379ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible/regions/62add0c2-3a4d-4471-bec1-2ae04c1e8e12
172025-09-19T20:47:02.379ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible/regions/62add0c2-3a4d-4471-bec1-2ae04c1e8e12
182025-09-19T20:47:02.490ZINFOcrucible-agent (worker): creating region Region { id: RegionId("62add0c2-3a4d-4471-bec1-2ae04c1e8e12"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/62add0c2-3a4d-4471-bec1-2ae04c1e8e12" region = 62add0c2-3a4d-4471-bec1-2ae04c1e8e12
192025-09-19T20:47:02.494ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "62add0c2-3a4d-4471-bec1-2ae04c1e8e12" "--data" "/data/regions/62add0c2-3a4d-4471-bec1-2ae04c1e8e12" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 62add0c2-3a4d-4471-bec1-2ae04c1e8e12
202025-09-19T20:47:02.562ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:60229 req_id = b2c9088c-44a9-4285-842c-26c759c3668e response_code = 200 uri = /crucible/0/regions
212025-09-19T20:47:02.904ZINFOcrucible-agent (worker): region files created ok region = 62add0c2-3a4d-4471-bec1-2ae04c1e8e12
222025-09-19T20:47:02.904ZINFOcrucible-agent (datafile): region 62add0c2-3a4d-4471-bec1-2ae04c1e8e12 state: Requested -> Created
232025-09-19T20:47:02.904ZINFOcrucible-agent (worker): applying SMF actions post create...
242025-09-19T20:47:02.905ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-62add0c2-3a4d-4471-bec1-2ae04c1e8e12
252025-09-19T20:47:02.917ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-62add0c2-3a4d-4471-bec1-2ae04c1e8e12
262025-09-19T20:47:02.946ZINFOcrucible-agent (worker): creating config property group
272025-09-19T20:47:02.956ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-62add0c2-3a4d-4471-bec1-2ae04c1e8e12
282025-09-19T20:47:02.956ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/62add0c2-3a4d-4471-bec1-2ae04c1e8e12
292025-09-19T20:47:02.966ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302025-09-19T20:47:02.966ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
312025-09-19T20:47:02.966ZINFOcrucible-agent (worker): commit
322025-09-19T20:47:02.991ZINFOcrucible-agent (worker): ok!
332025-09-19T20:47:03.000ZINFOcrucible-agent (worker): SMF ok!
342025-09-19T20:47:03.043ZINFOcrucible-agent (dropshot): request completed latency_us = 45 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:60229 req_id = 1520a142-6d2e-4cca-af34-9fe957da97c3 response_code = 200 uri = /crucible/0/regions
352025-09-19T20:47:04.019ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:60229 req_id = 1e390da4-199d-4a02-bfa1-a1c87315f06d response_code = 200 uri = /crucible/0/regions/62add0c2-3a4d-4471-bec1-2ae04c1e8e12
362025-09-19T20:47:04.086ZINFOcrucible-agent (datafile): path is "/data/regions/62add0c2-3a4d-4471-bec1-2ae04c1e8e12"
372025-09-19T20:47:04.098ZINFOcrucible-agent (datafile): dataset is oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible/regions/62add0c2-3a4d-4471-bec1-2ae04c1e8e12
382025-09-19T20:47:04.110ZINFOcrucible-agent (datafile): region 62add0c2-3a4d-4471-bec1-2ae04c1e8e12 state: Created -> Tombstoned
392025-09-19T20:47:04.110ZINFOcrucible-agent (worker): applying SMF actions before removal...
402025-09-19T20:47:04.110ZINFOcrucible-agent (dropshot): request completed latency_us = 24619 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:60229 req_id = 49311748-6e9d-4a0f-b364-1800a31e401c response_code = 204 uri = /crucible/0/regions/62add0c2-3a4d-4471-bec1-2ae04c1e8e12
412025-09-19T20:47:04.111ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-62add0c2-3a4d-4471-bec1-2ae04c1e8e12 (instance states: (Some(Online), None))
422025-09-19T20:47:04.114ZINFOcrucible-agent (dropshot): request completed latency_us = 28 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:60229 req_id = 5bd61f28-bf5a-4943-999f-2ab8b91834ab response_code = 200 uri = /crucible/0/regions/62add0c2-3a4d-4471-bec1-2ae04c1e8e12
432025-09-19T20:47:04.127ZINFOcrucible-agent (worker): SMF ok!
442025-09-19T20:47:04.140ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible/regions/62add0c2-3a4d-4471-bec1-2ae04c1e8e12" region = 62add0c2-3a4d-4471-bec1-2ae04c1e8e12
452025-09-19T20:47:04.187ZINFOcrucible-agent (dropshot): request completed latency_us = 82 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:60229 req_id = d3cfb815-d44a-482f-a7ad-d8bd72e881f2 response_code = 200 uri = /crucible/0/regions/62add0c2-3a4d-4471-bec1-2ae04c1e8e12
462025-09-19T20:47:04.261ZINFOcrucible-agent (datafile): region 62add0c2-3a4d-4471-bec1-2ae04c1e8e12 state: Tombstoned -> Destroyed
472025-09-19T20:47:04.340ZINFOcrucible-agent (dropshot): request completed latency_us = 37 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:60229 req_id = ffa9ce1e-b034-46df-b4a6-dfb1a5dbf651 response_code = 200 uri = /crucible/0/regions/62add0c2-3a4d-4471-bec1-2ae04c1e8e12
482025-09-19T20:47:07.137ZINFOcrucible-agent (datafile): region c9aa24c5-79ee-48f1-a37e-b81d46f9a725 state: Requested
492025-09-19T20:47:07.138ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
502025-09-19T20:47:07.138ZINFOcrucible-agent (dropshot): request completed latency_us = 199 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:60229 req_id = e31d1038-8c8a-40c1-af74-2a32cea49f85 response_code = 200 uri = /crucible/0/regions
512025-09-19T20:47:07.148ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725
522025-09-19T20:47:07.148ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725
532025-09-19T20:47:07.262ZINFOcrucible-agent (worker): creating region Region { id: RegionId("c9aa24c5-79ee-48f1-a37e-b81d46f9a725"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725" region = c9aa24c5-79ee-48f1-a37e-b81d46f9a725
542025-09-19T20:47:07.264ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "c9aa24c5-79ee-48f1-a37e-b81d46f9a725" "--data" "/data/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = c9aa24c5-79ee-48f1-a37e-b81d46f9a725
552025-09-19T20:47:07.410ZINFOcrucible-agent (worker): region files created ok region = c9aa24c5-79ee-48f1-a37e-b81d46f9a725
562025-09-19T20:47:07.410ZINFOcrucible-agent (datafile): region c9aa24c5-79ee-48f1-a37e-b81d46f9a725 state: Requested -> Created
572025-09-19T20:47:07.411ZINFOcrucible-agent (worker): applying SMF actions post create...
582025-09-19T20:47:07.412ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-c9aa24c5-79ee-48f1-a37e-b81d46f9a725
592025-09-19T20:47:07.424ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-c9aa24c5-79ee-48f1-a37e-b81d46f9a725
602025-09-19T20:47:07.444ZINFOcrucible-agent (worker): creating config property group
612025-09-19T20:47:07.454ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-c9aa24c5-79ee-48f1-a37e-b81d46f9a725
622025-09-19T20:47:07.454ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725
632025-09-19T20:47:07.454ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
642025-09-19T20:47:07.454ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
652025-09-19T20:47:07.454ZINFOcrucible-agent (worker): commit
662025-09-19T20:47:07.460ZINFOcrucible-agent (worker): ok!
672025-09-19T20:47:07.469ZINFOcrucible-agent (worker): SMF ok!
682025-09-19T20:47:07.504ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:60229 req_id = 34be3246-0362-477f-8c7d-fecab1db7e65 response_code = 200 uri = /crucible/0/regions
692025-09-19T20:57:50.343ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::b]:62981
702025-09-19T20:57:50.345ZINFOcrucible-agent (dropshot): request completed latency_us = 40 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:62981 req_id = 3cbac4e1-d2be-42de-bac6-b35b29b3c502 response_code = 200 uri = /crucible/0/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725
712025-09-19T20:57:50.349ZINFOcrucible-agent (datafile): path is "/data/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725"
722025-09-19T20:57:50.360ZINFOcrucible-agent (datafile): dataset is oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725
732025-09-19T20:57:50.369ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725@bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a
742025-09-19T20:57:50.369ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725", "bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a"]
752025-09-19T20:57:50.381ZINFOcrucible-agent (dropshot): request completed latency_us = 31544 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:62981 req_id = d906a03f-0b75-4b47-85ec-c760124a9fc7 response_code = 200 uri = /crucible/0/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725/snapshots/bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a
762025-09-19T20:57:50.388ZINFOcrucible-agent (datafile): path is "/data/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725"
772025-09-19T20:57:50.395ZINFOcrucible-agent (datafile): dataset is oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725
782025-09-19T20:57:50.405ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725@bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a
792025-09-19T20:57:50.405ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725", "bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a"]
802025-09-19T20:57:50.414ZINFOcrucible-agent (datafile): requesting running snapshot c9aa24c5-79ee-48f1-a37e-b81d46f9a725-bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a state: Requested
812025-09-19T20:57:50.414ZINFOcrucible-agent (worker): applying SMF actions for region c9aa24c5-79ee-48f1-a37e-b81d46f9a725 running snapshot bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a (state Requested)...
822025-09-19T20:57:50.414ZINFOcrucible-agent (dropshot): request completed latency_us = 26873 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:62981 req_id = 264121d4-467b-4fdb-8029-19b32a1cb759 response_code = 200 uri = /crucible/0/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725/snapshots/bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a/run
832025-09-19T20:57:50.428ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-c9aa24c5-79ee-48f1-a37e-b81d46f9a725-bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a
842025-09-19T20:57:50.443ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-c9aa24c5-79ee-48f1-a37e-b81d46f9a725-bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a
852025-09-19T20:57:50.467ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-c9aa24c5-79ee-48f1-a37e-b81d46f9a725-bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a
862025-09-19T20:57:50.467ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725/.zfs/snapshot/bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a
872025-09-19T20:57:50.467ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
882025-09-19T20:57:50.467ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
892025-09-19T20:57:50.467ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
902025-09-19T20:57:50.468ZINFOcrucible-agent (worker): commit
912025-09-19T20:57:50.474ZINFOcrucible-agent (worker): ok!
922025-09-19T20:57:50.482ZINFOcrucible-agent (worker): SMF ok!
932025-09-19T20:57:50.482ZINFOcrucible-agent (datafile): region c9aa24c5-79ee-48f1-a37e-b81d46f9a725 running snapshot bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a state: Requested -> Created
942025-09-19T20:57:50.583ZINFOcrucible-agent (datafile): path is "/data/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725"
952025-09-19T20:57:50.592ZINFOcrucible-agent (datafile): dataset is oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725
962025-09-19T20:57:50.600ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725@bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a
972025-09-19T20:57:50.600ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_cd1590c3-c6a6-4f65-bacb-62a193b0dd56/crucible/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725", "bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a"]
982025-09-19T20:57:50.608ZINFOcrucible-agent (dropshot): request completed latency_us = 25201 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:62981 req_id = 2a51510a-2306-4089-9541-621196ce2a16 response_code = 200 uri = /crucible/0/regions/c9aa24c5-79ee-48f1-a37e-b81d46f9a725/snapshots/bddf0dcb-bd19-4f2c-8c1b-3b9508c0755a/run