Max level shown:
1 [ May 19 16:47:42 Enabled. ]
2 [ May 19 16:47:42 Rereading configuration. ]
3 [ May 19 16:47:43 Rereading configuration. ]
4 [ May 19 16:47:47 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible -l [fd00:1122:3344:101::12]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-05-19T16:47:47.270ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 7103cd3a3d7b0112d2949dd135db06fef0c156bb Commit timestamp: 2025-11-10T22:24:53.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-05-19T16:47:47.270ZINFOcrucible-agent: dataset: "oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible"
82026-05-19T16:47:47.270ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::12]:32345
92026-05-19T16:47:47.270ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-05-19T16:47:47.284ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-05-19T16:47:47.360ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::12]:32345
122026-05-19T16:48:22.888ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:35696
132026-05-19T16:48:22.913ZINFOcrucible-agent (datafile): region 05ad1ec0-d9a7-4305-8019-7a13ae60f1ae state: Requested
142026-05-19T16:48:22.917ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152026-05-19T16:48:22.917ZINFOcrucible-agent (dropshot): request completed latency_us = 13885 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:35696 req_id = 05ced395-e9b2-4020-97cc-a9b6f0e40e0d response_code = 200 uri = /crucible/0/regions
162026-05-19T16:48:22.924ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible/regions/05ad1ec0-d9a7-4305-8019-7a13ae60f1ae
172026-05-19T16:48:22.924ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible/regions/05ad1ec0-d9a7-4305-8019-7a13ae60f1ae
182026-05-19T16:48:23.002ZINFOcrucible-agent (worker): creating region Region { id: RegionId("05ad1ec0-d9a7-4305-8019-7a13ae60f1ae"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/05ad1ec0-d9a7-4305-8019-7a13ae60f1ae" region = 05ad1ec0-d9a7-4305-8019-7a13ae60f1ae
192026-05-19T16:48:23.004ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "05ad1ec0-d9a7-4305-8019-7a13ae60f1ae" "--data" "/data/regions/05ad1ec0-d9a7-4305-8019-7a13ae60f1ae" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 05ad1ec0-d9a7-4305-8019-7a13ae60f1ae
202026-05-19T16:48:23.184ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:35696 req_id = f60f1b78-488b-4b2c-831d-0ecc450b49fa response_code = 200 uri = /crucible/0/regions
212026-05-19T16:48:23.380ZINFOcrucible-agent (worker): region files created ok region = 05ad1ec0-d9a7-4305-8019-7a13ae60f1ae
222026-05-19T16:48:23.380ZINFOcrucible-agent (datafile): region 05ad1ec0-d9a7-4305-8019-7a13ae60f1ae state: Requested -> Created
232026-05-19T16:48:23.380ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-05-19T16:48:23.381ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-05ad1ec0-d9a7-4305-8019-7a13ae60f1ae
252026-05-19T16:48:23.392ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-05ad1ec0-d9a7-4305-8019-7a13ae60f1ae
262026-05-19T16:48:23.415ZINFOcrucible-agent (worker): creating config property group
272026-05-19T16:48:23.423ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-05ad1ec0-d9a7-4305-8019-7a13ae60f1ae
282026-05-19T16:48:23.424ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/05ad1ec0-d9a7-4305-8019-7a13ae60f1ae
292026-05-19T16:48:23.424ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-05-19T16:48:23.424ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
312026-05-19T16:48:23.424ZINFOcrucible-agent (worker): commit
322026-05-19T16:48:23.431ZINFOcrucible-agent (worker): ok!
332026-05-19T16:48:23.438ZINFOcrucible-agent (worker): SMF ok!
342026-05-19T16:48:23.525ZINFOcrucible-agent (dropshot): request completed latency_us = 63 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:35696 req_id = 7bdd64db-95f8-44bb-a491-0d8d8a72d55e response_code = 200 uri = /crucible/0/regions
352026-05-19T16:48:24.268ZINFOcrucible-agent (dropshot): request completed latency_us = 94 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:35696 req_id = e363049f-8fe3-4629-8bcb-f5ebc4200e54 response_code = 200 uri = /crucible/0/regions/05ad1ec0-d9a7-4305-8019-7a13ae60f1ae
362026-05-19T16:48:24.275ZINFOcrucible-agent (datafile): path is "/data/regions/05ad1ec0-d9a7-4305-8019-7a13ae60f1ae"
372026-05-19T16:48:24.283ZINFOcrucible-agent (datafile): dataset is oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible/regions/05ad1ec0-d9a7-4305-8019-7a13ae60f1ae
382026-05-19T16:48:24.290ZINFOcrucible-agent (datafile): region 05ad1ec0-d9a7-4305-8019-7a13ae60f1ae state: Created -> Tombstoned
392026-05-19T16:48:24.290ZINFOcrucible-agent (worker): applying SMF actions before removal...
402026-05-19T16:48:24.290ZINFOcrucible-agent (dropshot): request completed latency_us = 15178 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:35696 req_id = 7ed355d8-5a50-486c-a304-4385d29fb006 response_code = 204 uri = /crucible/0/regions/05ad1ec0-d9a7-4305-8019-7a13ae60f1ae
412026-05-19T16:48:24.291ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-05ad1ec0-d9a7-4305-8019-7a13ae60f1ae (instance states: (Some(Online), None))
422026-05-19T16:48:24.293ZINFOcrucible-agent (dropshot): request completed latency_us = 24 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:35696 req_id = fc79d3dc-4a87-4273-8c80-79b09ac429ab response_code = 200 uri = /crucible/0/regions/05ad1ec0-d9a7-4305-8019-7a13ae60f1ae
432026-05-19T16:48:24.298ZINFOcrucible-agent (worker): SMF ok!
442026-05-19T16:48:24.305ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible/regions/05ad1ec0-d9a7-4305-8019-7a13ae60f1ae" region = 05ad1ec0-d9a7-4305-8019-7a13ae60f1ae
452026-05-19T16:48:24.382ZINFOcrucible-agent (datafile): region 05ad1ec0-d9a7-4305-8019-7a13ae60f1ae state: Tombstoned -> Destroyed
462026-05-19T16:48:24.425ZINFOcrucible-agent (dropshot): request completed latency_us = 51 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:35696 req_id = 72653002-9ee1-4d9d-8ecf-086475bacf41 response_code = 200 uri = /crucible/0/regions/05ad1ec0-d9a7-4305-8019-7a13ae60f1ae
472026-05-19T16:48:26.863ZINFOcrucible-agent (datafile): region 0c5232bf-6020-4f9e-99fd-7dd6a24747de state: Requested
482026-05-19T16:48:26.863ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
492026-05-19T16:48:26.863ZINFOcrucible-agent (dropshot): request completed latency_us = 260 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:35696 req_id = 4422257e-8a3d-4d07-87e8-4cb513cea724 response_code = 200 uri = /crucible/0/regions
502026-05-19T16:48:26.870ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de
512026-05-19T16:48:26.870ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de
522026-05-19T16:48:26.959ZINFOcrucible-agent (worker): creating region Region { id: RegionId("0c5232bf-6020-4f9e-99fd-7dd6a24747de"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de" region = 0c5232bf-6020-4f9e-99fd-7dd6a24747de
532026-05-19T16:48:26.959ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "0c5232bf-6020-4f9e-99fd-7dd6a24747de" "--data" "/data/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 0c5232bf-6020-4f9e-99fd-7dd6a24747de
542026-05-19T16:48:27.017ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:35696 req_id = 53ccf553-6da8-4cef-8f79-d54c37a020b4 response_code = 200 uri = /crucible/0/regions
552026-05-19T16:48:27.094ZINFOcrucible-agent (worker): region files created ok region = 0c5232bf-6020-4f9e-99fd-7dd6a24747de
562026-05-19T16:48:27.094ZINFOcrucible-agent (datafile): region 0c5232bf-6020-4f9e-99fd-7dd6a24747de state: Requested -> Created
572026-05-19T16:48:27.094ZINFOcrucible-agent (worker): applying SMF actions post create...
582026-05-19T16:48:27.096ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-0c5232bf-6020-4f9e-99fd-7dd6a24747de
592026-05-19T16:48:27.101ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-0c5232bf-6020-4f9e-99fd-7dd6a24747de
602026-05-19T16:48:27.112ZINFOcrucible-agent (worker): creating config property group
612026-05-19T16:48:27.115ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0c5232bf-6020-4f9e-99fd-7dd6a24747de
622026-05-19T16:48:27.116ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de
632026-05-19T16:48:27.116ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
642026-05-19T16:48:27.116ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
652026-05-19T16:48:27.116ZINFOcrucible-agent (worker): commit
662026-05-19T16:48:27.122ZINFOcrucible-agent (worker): ok!
672026-05-19T16:48:27.129ZINFOcrucible-agent (worker): SMF ok!
682026-05-19T16:48:27.652ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:35696 req_id = 3d827586-e176-46a1-80cd-85b7bb3a109d response_code = 200 uri = /crucible/0/regions
692026-05-19T16:59:19.364ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:42177
702026-05-19T16:59:19.365ZINFOcrucible-agent (dropshot): request completed latency_us = 40 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42177 req_id = 47c77568-1e12-49d9-9814-1baa2e2fc5d1 response_code = 200 uri = /crucible/0/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de
712026-05-19T16:59:19.368ZINFOcrucible-agent (datafile): path is "/data/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de"
722026-05-19T16:59:19.375ZINFOcrucible-agent (datafile): dataset is oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de
732026-05-19T16:59:19.381ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de@f59f1dc0-742a-47ec-9a38-6a3c7cc825cf
742026-05-19T16:59:19.381ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de", "f59f1dc0-742a-47ec-9a38-6a3c7cc825cf"]
752026-05-19T16:59:19.392ZINFOcrucible-agent (dropshot): request completed latency_us = 23904 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42177 req_id = 79e271a7-cfa2-42c3-b3d9-f0d136654a1f response_code = 200 uri = /crucible/0/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de/snapshots/f59f1dc0-742a-47ec-9a38-6a3c7cc825cf
762026-05-19T16:59:19.395ZINFOcrucible-agent (datafile): path is "/data/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de"
772026-05-19T16:59:19.401ZINFOcrucible-agent (datafile): dataset is oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de
782026-05-19T16:59:19.407ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de@f59f1dc0-742a-47ec-9a38-6a3c7cc825cf
792026-05-19T16:59:19.407ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de", "f59f1dc0-742a-47ec-9a38-6a3c7cc825cf"]
802026-05-19T16:59:19.414ZINFOcrucible-agent (datafile): requesting running snapshot 0c5232bf-6020-4f9e-99fd-7dd6a24747de-f59f1dc0-742a-47ec-9a38-6a3c7cc825cf state: Requested
812026-05-19T16:59:19.415ZINFOcrucible-agent (worker): applying SMF actions for region 0c5232bf-6020-4f9e-99fd-7dd6a24747de running snapshot f59f1dc0-742a-47ec-9a38-6a3c7cc825cf (state Requested)...
822026-05-19T16:59:19.415ZINFOcrucible-agent (dropshot): request completed latency_us = 19878 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42177 req_id = 19638389-9670-43c2-9bda-f401be61056e response_code = 200 uri = /crucible/0/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de/snapshots/f59f1dc0-742a-47ec-9a38-6a3c7cc825cf/run
832026-05-19T16:59:19.437ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-0c5232bf-6020-4f9e-99fd-7dd6a24747de-f59f1dc0-742a-47ec-9a38-6a3c7cc825cf
842026-05-19T16:59:19.448ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-0c5232bf-6020-4f9e-99fd-7dd6a24747de-f59f1dc0-742a-47ec-9a38-6a3c7cc825cf
852026-05-19T16:59:19.463ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-0c5232bf-6020-4f9e-99fd-7dd6a24747de-f59f1dc0-742a-47ec-9a38-6a3c7cc825cf
862026-05-19T16:59:19.463ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de/.zfs/snapshot/f59f1dc0-742a-47ec-9a38-6a3c7cc825cf
872026-05-19T16:59:19.463ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
882026-05-19T16:59:19.463ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
892026-05-19T16:59:19.463ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
902026-05-19T16:59:19.463ZINFOcrucible-agent (worker): commit
912026-05-19T16:59:19.467ZINFOcrucible-agent (worker): ok!
922026-05-19T16:59:19.471ZINFOcrucible-agent (worker): SMF ok!
932026-05-19T16:59:19.471ZINFOcrucible-agent (datafile): region 0c5232bf-6020-4f9e-99fd-7dd6a24747de running snapshot f59f1dc0-742a-47ec-9a38-6a3c7cc825cf state: Requested -> Created
942026-05-19T16:59:19.693ZINFOcrucible-agent (datafile): path is "/data/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de"
952026-05-19T16:59:19.700ZINFOcrucible-agent (datafile): dataset is oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de
962026-05-19T16:59:19.707ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de@f59f1dc0-742a-47ec-9a38-6a3c7cc825cf
972026-05-19T16:59:19.707ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_3dc58145-d6fc-4a90-ba5b-4331dcafaecc/crucible/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de", "f59f1dc0-742a-47ec-9a38-6a3c7cc825cf"]
982026-05-19T16:59:19.714ZINFOcrucible-agent (dropshot): request completed latency_us = 20463 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42177 req_id = 3659aaff-bed2-4dbf-ae9e-609799a4785a response_code = 200 uri = /crucible/0/regions/0c5232bf-6020-4f9e-99fd-7dd6a24747de/snapshots/f59f1dc0-742a-47ec-9a38-6a3c7cc825cf/run