Max level shown:
1 [ Aug 30 14:27:40 Enabled. ]
2 [ Aug 30 14:27:40 Rereading configuration. ]
3 [ Aug 30 14:27:43 Rereading configuration. ]
4 [ Aug 30 14:27:48 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_45c4c681-a71e-4acf-9b71-633152c825cf/crucible -l [fd00:1122:3344:101::12]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-08-30T14:27:48.709ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 779775d5130ff7a4836f52f48b7e64d1479ee104 Commit timestamp: 2025-08-09T00:27:59.000000000Z branch: main rustc: 1.86.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72025-08-30T14:27:48.712ZINFOcrucible-agent: dataset: "oxp_45c4c681-a71e-4acf-9b71-633152c825cf/crucible"
82025-08-30T14:27:48.712ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::12]:32345
92025-08-30T14:27:48.712ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102025-08-30T14:27:48.728ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112025-08-30T14:27:48.833ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::12]:32345
122025-08-30T14:28:27.254ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::b]:42315
132025-08-30T14:28:27.296ZINFOcrucible-agent (datafile): region caeb4974-0a29-4b21-92cd-dff744abb355 state: Requested
142025-08-30T14:28:27.303ZINFOcrucible-agent (dropshot): request completed latency_us = 24384 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:42315 req_id = e6270bb9-7255-43c0-925b-053d21b58bd2 response_code = 200 uri = /crucible/0/regions
152025-08-30T14:28:27.308ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
162025-08-30T14:28:27.318ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_45c4c681-a71e-4acf-9b71-633152c825cf/crucible/regions/caeb4974-0a29-4b21-92cd-dff744abb355
172025-08-30T14:28:27.318ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_45c4c681-a71e-4acf-9b71-633152c825cf/crucible/regions/caeb4974-0a29-4b21-92cd-dff744abb355
182025-08-30T14:28:27.387ZINFOcrucible-agent (worker): creating region Region { id: RegionId("caeb4974-0a29-4b21-92cd-dff744abb355"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/caeb4974-0a29-4b21-92cd-dff744abb355" region = caeb4974-0a29-4b21-92cd-dff744abb355
192025-08-30T14:28:27.389ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "caeb4974-0a29-4b21-92cd-dff744abb355" "--data" "/data/regions/caeb4974-0a29-4b21-92cd-dff744abb355" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = caeb4974-0a29-4b21-92cd-dff744abb355
202025-08-30T14:28:27.528ZINFOcrucible-agent (dropshot): request completed latency_us = 89 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:42315 req_id = 4e6489ec-04f4-41d1-ad22-c2972b4dee92 response_code = 200 uri = /crucible/0/regions
212025-08-30T14:28:27.664ZINFOcrucible-agent (worker): region files created ok region = caeb4974-0a29-4b21-92cd-dff744abb355
222025-08-30T14:28:27.664ZINFOcrucible-agent (datafile): region caeb4974-0a29-4b21-92cd-dff744abb355 state: Requested -> Created
232025-08-30T14:28:27.665ZINFOcrucible-agent (worker): applying SMF actions post create...
242025-08-30T14:28:27.665ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-caeb4974-0a29-4b21-92cd-dff744abb355
252025-08-30T14:28:27.675ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-caeb4974-0a29-4b21-92cd-dff744abb355
262025-08-30T14:28:27.703ZINFOcrucible-agent (worker): creating config property group
272025-08-30T14:28:27.708ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-caeb4974-0a29-4b21-92cd-dff744abb355
282025-08-30T14:28:27.708ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/caeb4974-0a29-4b21-92cd-dff744abb355
292025-08-30T14:28:27.718ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302025-08-30T14:28:27.719ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
312025-08-30T14:28:27.719ZINFOcrucible-agent (worker): commit
322025-08-30T14:28:27.724ZINFOcrucible-agent (worker): ok!
332025-08-30T14:28:27.729ZINFOcrucible-agent (worker): SMF ok!
342025-08-30T14:28:28.059ZINFOcrucible-agent (dropshot): request completed latency_us = 45 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:42315 req_id = 8381a52b-a550-410e-842e-a8fe04370f27 response_code = 200 uri = /crucible/0/regions
352025-08-30T14:28:28.635ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:42315 req_id = 3789046c-3708-439b-a4b0-e0d6109eeee9 response_code = 200 uri = /crucible/0/regions/caeb4974-0a29-4b21-92cd-dff744abb355
362025-08-30T14:28:28.639ZINFOcrucible-agent (datafile): path is "/data/regions/caeb4974-0a29-4b21-92cd-dff744abb355"
372025-08-30T14:28:28.652ZINFOcrucible-agent (datafile): dataset is oxp_45c4c681-a71e-4acf-9b71-633152c825cf/crucible/regions/caeb4974-0a29-4b21-92cd-dff744abb355
382025-08-30T14:28:28.664ZINFOcrucible-agent (datafile): region caeb4974-0a29-4b21-92cd-dff744abb355 state: Created -> Tombstoned
392025-08-30T14:28:28.665ZINFOcrucible-agent (worker): applying SMF actions before removal...
402025-08-30T14:28:28.665ZINFOcrucible-agent (dropshot): request completed latency_us = 25921 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:42315 req_id = a62eea86-7989-4310-8aa4-9fd55c670e60 response_code = 204 uri = /crucible/0/regions/caeb4974-0a29-4b21-92cd-dff744abb355
412025-08-30T14:28:28.666ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-caeb4974-0a29-4b21-92cd-dff744abb355 (instance states: (Some(Online), None))
422025-08-30T14:28:28.669ZINFOcrucible-agent (dropshot): request completed latency_us = 32 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:42315 req_id = 7ce80a7b-3bca-464a-8877-778d7d55c7ce response_code = 200 uri = /crucible/0/regions/caeb4974-0a29-4b21-92cd-dff744abb355
432025-08-30T14:28:28.670ZINFOcrucible-agent (worker): SMF ok!
442025-08-30T14:28:28.683ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_45c4c681-a71e-4acf-9b71-633152c825cf/crucible/regions/caeb4974-0a29-4b21-92cd-dff744abb355" region = caeb4974-0a29-4b21-92cd-dff744abb355
452025-08-30T14:28:28.759ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:42315 req_id = f7eeb1bc-83c6-4291-a054-8fa135b005aa response_code = 200 uri = /crucible/0/regions/caeb4974-0a29-4b21-92cd-dff744abb355
462025-08-30T14:28:28.762ZINFOcrucible-agent (datafile): region caeb4974-0a29-4b21-92cd-dff744abb355 state: Tombstoned -> Destroyed
472025-08-30T14:28:28.874ZINFOcrucible-agent (dropshot): request completed latency_us = 41 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:42315 req_id = b3d553e3-c92a-48ab-9962-43349976ca0c response_code = 200 uri = /crucible/0/regions/caeb4974-0a29-4b21-92cd-dff744abb355
482025-08-30T14:28:31.266ZINFOcrucible-agent (datafile): region 2ca978dc-853a-4c7a-b110-c3efb9d11675 state: Requested
492025-08-30T14:28:31.266ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
502025-08-30T14:28:31.267ZINFOcrucible-agent (dropshot): request completed latency_us = 203 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:42315 req_id = 9f030f68-0d0b-42a9-9a2c-81195f6105ef response_code = 200 uri = /crucible/0/regions
512025-08-30T14:28:31.277ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_45c4c681-a71e-4acf-9b71-633152c825cf/crucible/regions/2ca978dc-853a-4c7a-b110-c3efb9d11675
522025-08-30T14:28:31.277ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_45c4c681-a71e-4acf-9b71-633152c825cf/crucible/regions/2ca978dc-853a-4c7a-b110-c3efb9d11675
532025-08-30T14:28:31.343ZINFOcrucible-agent (worker): creating region Region { id: RegionId("2ca978dc-853a-4c7a-b110-c3efb9d11675"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/2ca978dc-853a-4c7a-b110-c3efb9d11675" region = 2ca978dc-853a-4c7a-b110-c3efb9d11675
542025-08-30T14:28:31.343ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "2ca978dc-853a-4c7a-b110-c3efb9d11675" "--data" "/data/regions/2ca978dc-853a-4c7a-b110-c3efb9d11675" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 2ca978dc-853a-4c7a-b110-c3efb9d11675
552025-08-30T14:28:31.428ZINFOcrucible-agent (worker): region files created ok region = 2ca978dc-853a-4c7a-b110-c3efb9d11675
562025-08-30T14:28:31.428ZINFOcrucible-agent (datafile): region 2ca978dc-853a-4c7a-b110-c3efb9d11675 state: Requested -> Created
572025-08-30T14:28:31.428ZINFOcrucible-agent (worker): applying SMF actions post create...
582025-08-30T14:28:31.429ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-2ca978dc-853a-4c7a-b110-c3efb9d11675
592025-08-30T14:28:31.437ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-2ca978dc-853a-4c7a-b110-c3efb9d11675
602025-08-30T14:28:31.448ZINFOcrucible-agent (worker): creating config property group
612025-08-30T14:28:31.451ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-2ca978dc-853a-4c7a-b110-c3efb9d11675
622025-08-30T14:28:31.452ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/2ca978dc-853a-4c7a-b110-c3efb9d11675
632025-08-30T14:28:31.452ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
642025-08-30T14:28:31.452ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
652025-08-30T14:28:31.452ZINFOcrucible-agent (worker): commit
662025-08-30T14:28:31.458ZINFOcrucible-agent (worker): ok!
672025-08-30T14:28:31.463ZINFOcrucible-agent (worker): SMF ok!
682025-08-30T14:28:31.535ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:42315 req_id = 328343ca-8915-4b90-b8e8-3e103bfc2641 response_code = 200 uri = /crucible/0/regions