Max level shown:
1 [ Feb 23 16:30:44 Enabled. ]
2 [ Feb 23 16:30:44 Rereading configuration. ]
3 [ Feb 23 16:30:47 Rereading configuration. ]
4 [ Feb 23 16:30:55 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible -l [fd00:1122:3344:101::12]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-02-23T16:30:56.041ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 7103cd3a3d7b0112d2949dd135db06fef0c156bb Commit timestamp: 2025-11-10T22:24:53.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-02-23T16:30:56.041ZINFOcrucible-agent: dataset: "oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible"
82026-02-23T16:30:56.041ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::12]:32345
92026-02-23T16:30:56.041ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-02-23T16:30:56.055ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-02-23T16:30:56.246ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::12]:32345
122026-02-23T16:31:50.463ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::c]:48692
132026-02-23T16:31:50.491ZINFOcrucible-agent (datafile): region 140bd3f8-89b0-4d23-8f99-c66e9dc950cf state: Requested
142026-02-23T16:31:50.494ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152026-02-23T16:31:50.494ZINFOcrucible-agent (dropshot): request completed latency_us = 16561 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:48692 req_id = ff6fbe8c-e3bd-40fd-b8a3-be52281aa4ba response_code = 200 uri = /crucible/0/regions
162026-02-23T16:31:50.502ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf
172026-02-23T16:31:50.502ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf
182026-02-23T16:31:50.581ZINFOcrucible-agent (worker): creating region Region { id: RegionId("140bd3f8-89b0-4d23-8f99-c66e9dc950cf"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf" region = 140bd3f8-89b0-4d23-8f99-c66e9dc950cf
192026-02-23T16:31:50.583ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "140bd3f8-89b0-4d23-8f99-c66e9dc950cf" "--data" "/data/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 140bd3f8-89b0-4d23-8f99-c66e9dc950cf
202026-02-23T16:31:50.662ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:48692 req_id = ee8feeb9-8f58-41a4-8637-328592ff2fa9 response_code = 200 uri = /crucible/0/regions
212026-02-23T16:31:50.957ZINFOcrucible-agent (worker): region files created ok region = 140bd3f8-89b0-4d23-8f99-c66e9dc950cf
222026-02-23T16:31:50.957ZINFOcrucible-agent (datafile): region 140bd3f8-89b0-4d23-8f99-c66e9dc950cf state: Requested -> Created
232026-02-23T16:31:50.957ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-02-23T16:31:50.957ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-140bd3f8-89b0-4d23-8f99-c66e9dc950cf
252026-02-23T16:31:50.964ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-140bd3f8-89b0-4d23-8f99-c66e9dc950cf
262026-02-23T16:31:50.977ZINFOcrucible-agent (worker): creating config property group
272026-02-23T16:31:50.981ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-140bd3f8-89b0-4d23-8f99-c66e9dc950cf
282026-02-23T16:31:50.981ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf
292026-02-23T16:31:50.981ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-02-23T16:31:50.981ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
312026-02-23T16:31:50.981ZINFOcrucible-agent (worker): commit
322026-02-23T16:31:50.986ZINFOcrucible-agent (worker): ok!
332026-02-23T16:31:50.991ZINFOcrucible-agent (worker): SMF ok!
342026-02-23T16:31:51.120ZINFOcrucible-agent (dropshot): request completed latency_us = 63 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:48692 req_id = 187e3373-730a-4783-809a-716553ed2d50 response_code = 200 uri = /crucible/0/regions
352026-02-23T16:43:17.491ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::c]:62285
362026-02-23T16:43:17.491ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:62285 req_id = 07742961-7def-4268-8322-1568ef3d4aeb response_code = 200 uri = /crucible/0/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf
372026-02-23T16:43:17.494ZINFOcrucible-agent (datafile): path is "/data/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf"
382026-02-23T16:43:17.499ZINFOcrucible-agent (datafile): dataset is oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf
392026-02-23T16:43:17.504ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf@501ad197-e2ed-4849-8d91-14222bfc9a60
402026-02-23T16:43:17.504ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf", "501ad197-e2ed-4849-8d91-14222bfc9a60"]
412026-02-23T16:43:17.513ZINFOcrucible-agent (dropshot): request completed latency_us = 18745 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:62285 req_id = 10cb4440-d295-4207-bfa4-1bfae366df92 response_code = 200 uri = /crucible/0/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf/snapshots/501ad197-e2ed-4849-8d91-14222bfc9a60
422026-02-23T16:43:17.516ZINFOcrucible-agent (datafile): path is "/data/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf"
432026-02-23T16:43:17.521ZINFOcrucible-agent (datafile): dataset is oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf
442026-02-23T16:43:17.526ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf@501ad197-e2ed-4849-8d91-14222bfc9a60
452026-02-23T16:43:17.526ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf", "501ad197-e2ed-4849-8d91-14222bfc9a60"]
462026-02-23T16:43:17.532ZINFOcrucible-agent (datafile): requesting running snapshot 140bd3f8-89b0-4d23-8f99-c66e9dc950cf-501ad197-e2ed-4849-8d91-14222bfc9a60 state: Requested
472026-02-23T16:43:17.532ZINFOcrucible-agent (dropshot): request completed latency_us = 16607 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:62285 req_id = ca9191eb-abeb-491e-8077-23f69fedb1b0 response_code = 200 uri = /crucible/0/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf/snapshots/501ad197-e2ed-4849-8d91-14222bfc9a60/run
482026-02-23T16:43:17.532ZINFOcrucible-agent (worker): applying SMF actions for region 140bd3f8-89b0-4d23-8f99-c66e9dc950cf running snapshot 501ad197-e2ed-4849-8d91-14222bfc9a60 (state Requested)...
492026-02-23T16:43:17.554ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-140bd3f8-89b0-4d23-8f99-c66e9dc950cf-501ad197-e2ed-4849-8d91-14222bfc9a60
502026-02-23T16:43:17.561ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-140bd3f8-89b0-4d23-8f99-c66e9dc950cf-501ad197-e2ed-4849-8d91-14222bfc9a60
512026-02-23T16:43:17.574ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-140bd3f8-89b0-4d23-8f99-c66e9dc950cf-501ad197-e2ed-4849-8d91-14222bfc9a60
522026-02-23T16:43:17.574ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf/.zfs/snapshot/501ad197-e2ed-4849-8d91-14222bfc9a60
532026-02-23T16:43:17.574ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
542026-02-23T16:43:17.574ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
552026-02-23T16:43:17.574ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
562026-02-23T16:43:17.574ZINFOcrucible-agent (worker): commit
572026-02-23T16:43:17.578ZINFOcrucible-agent (worker): ok!
582026-02-23T16:43:17.583ZINFOcrucible-agent (worker): SMF ok!
592026-02-23T16:43:17.583ZINFOcrucible-agent (datafile): region 140bd3f8-89b0-4d23-8f99-c66e9dc950cf running snapshot 501ad197-e2ed-4849-8d91-14222bfc9a60 state: Requested -> Created
602026-02-23T16:43:17.762ZINFOcrucible-agent (datafile): path is "/data/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf"
612026-02-23T16:43:17.768ZINFOcrucible-agent (datafile): dataset is oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf
622026-02-23T16:43:17.773ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf@501ad197-e2ed-4849-8d91-14222bfc9a60
632026-02-23T16:43:17.773ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf", "501ad197-e2ed-4849-8d91-14222bfc9a60"]
642026-02-23T16:43:17.779ZINFOcrucible-agent (dropshot): request completed latency_us = 17622 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:62285 req_id = d29387e6-2963-4786-a11f-cf6ff5d6b478 response_code = 200 uri = /crucible/0/regions/140bd3f8-89b0-4d23-8f99-c66e9dc950cf/snapshots/501ad197-e2ed-4849-8d91-14222bfc9a60/run
652026-02-23T16:43:22.752ZINFOcrucible-agent (datafile): region 8f4b4d1f-a1e8-4739-bffd-1d5d588ae269 state: Requested
662026-02-23T16:43:22.753ZINFOcrucible-agent (dropshot): request completed latency_us = 221 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:62285 req_id = 6e7b82cd-7c2b-4284-a69d-f7a46309dd4f response_code = 200 uri = /crucible/0/regions
672026-02-23T16:43:22.753ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
682026-02-23T16:43:22.759ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible/regions/8f4b4d1f-a1e8-4739-bffd-1d5d588ae269
692026-02-23T16:43:22.759ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible/regions/8f4b4d1f-a1e8-4739-bffd-1d5d588ae269
702026-02-23T16:43:22.814ZINFOcrucible-agent (worker): creating region Region { id: RegionId("8f4b4d1f-a1e8-4739-bffd-1d5d588ae269"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/8f4b4d1f-a1e8-4739-bffd-1d5d588ae269" region = 8f4b4d1f-a1e8-4739-bffd-1d5d588ae269
712026-02-23T16:43:22.814ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "8f4b4d1f-a1e8-4739-bffd-1d5d588ae269" "--data" "/data/regions/8f4b4d1f-a1e8-4739-bffd-1d5d588ae269" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 8f4b4d1f-a1e8-4739-bffd-1d5d588ae269
722026-02-23T16:43:22.905ZINFOcrucible-agent (worker): region files created ok region = 8f4b4d1f-a1e8-4739-bffd-1d5d588ae269
732026-02-23T16:43:22.905ZINFOcrucible-agent (datafile): region 8f4b4d1f-a1e8-4739-bffd-1d5d588ae269 state: Requested -> Created
742026-02-23T16:43:22.905ZINFOcrucible-agent (worker): applying SMF actions post create...
752026-02-23T16:43:22.906ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-8f4b4d1f-a1e8-4739-bffd-1d5d588ae269
762026-02-23T16:43:22.918ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-8f4b4d1f-a1e8-4739-bffd-1d5d588ae269
772026-02-23T16:43:22.925ZINFOcrucible-agent (worker): creating config property group
782026-02-23T16:43:22.929ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-8f4b4d1f-a1e8-4739-bffd-1d5d588ae269
792026-02-23T16:43:22.929ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/8f4b4d1f-a1e8-4739-bffd-1d5d588ae269
802026-02-23T16:43:22.929ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
812026-02-23T16:43:22.929ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
822026-02-23T16:43:22.929ZINFOcrucible-agent (worker): commit
832026-02-23T16:43:22.932ZINFOcrucible-agent (worker): ok!
842026-02-23T16:43:22.938ZINFOcrucible-agent (worker): SMF ok!
852026-02-23T16:43:23.026ZINFOcrucible-agent (dropshot): request completed latency_us = 63 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:62285 req_id = ade726e7-e561-43c1-abca-b06822468e5b response_code = 200 uri = /crucible/0/regions
862026-02-23T16:43:47.423ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:62285 req_id = 45b46c79-51ac-4661-8cde-7bb2d472158d response_code = 200 uri = /crucible/0/regions/8f4b4d1f-a1e8-4739-bffd-1d5d588ae269
872026-02-23T16:43:47.426ZINFOcrucible-agent (datafile): path is "/data/regions/8f4b4d1f-a1e8-4739-bffd-1d5d588ae269"
882026-02-23T16:43:47.435ZINFOcrucible-agent (datafile): dataset is oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible/regions/8f4b4d1f-a1e8-4739-bffd-1d5d588ae269
892026-02-23T16:43:47.442ZINFOcrucible-agent (datafile): region 8f4b4d1f-a1e8-4739-bffd-1d5d588ae269 state: Created -> Tombstoned
902026-02-23T16:43:47.443ZINFOcrucible-agent (dropshot): request completed latency_us = 17329 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:62285 req_id = e6f6840f-620f-42d7-9623-39bf3f96a815 response_code = 204 uri = /crucible/0/regions/8f4b4d1f-a1e8-4739-bffd-1d5d588ae269
912026-02-23T16:43:47.443ZINFOcrucible-agent (worker): applying SMF actions before removal...
922026-02-23T16:43:47.445ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-8f4b4d1f-a1e8-4739-bffd-1d5d588ae269 (instance states: (Some(Online), None))
932026-02-23T16:43:47.446ZINFOcrucible-agent (dropshot): request completed latency_us = 51 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:62285 req_id = c8748057-f548-4ca4-80fb-9469784efbea response_code = 200 uri = /crucible/0/regions/8f4b4d1f-a1e8-4739-bffd-1d5d588ae269
942026-02-23T16:43:47.459ZINFOcrucible-agent (worker): SMF ok!
952026-02-23T16:43:47.466ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_19a7746a-6f34-4bcd-bd11-3581a9471156/crucible/regions/8f4b4d1f-a1e8-4739-bffd-1d5d588ae269" region = 8f4b4d1f-a1e8-4739-bffd-1d5d588ae269
962026-02-23T16:43:47.503ZINFOcrucible-agent (dropshot): request completed latency_us = 42 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:62285 req_id = 77561dac-6a06-4124-8c66-e3e7c9fbe76d response_code = 200 uri = /crucible/0/regions/8f4b4d1f-a1e8-4739-bffd-1d5d588ae269
972026-02-23T16:43:47.530ZINFOcrucible-agent (datafile): region 8f4b4d1f-a1e8-4739-bffd-1d5d588ae269 state: Tombstoned -> Destroyed
982026-02-23T16:43:47.668ZINFOcrucible-agent (dropshot): request completed latency_us = 33 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:62285 req_id = 02d90a95-6805-4b1d-948c-b70131e48fb5 response_code = 200 uri = /crucible/0/regions/8f4b4d1f-a1e8-4739-bffd-1d5d588ae269