Max level shown:
1 [ Feb 11 20:45:01 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_d57f35fc-ef0a-4e26-8a23-712fdd5557d6/crucible -l [fd00:1122:3344:101::17]:32345 -P 19000 -p downstairs -s snapshot"). ]
2 note: configured to log to "/dev/stdout"
32025-02-11T20:45:01.843ZINFOcrucible-agent: dataset: "oxp_d57f35fc-ef0a-4e26-8a23-712fdd5557d6/crucible"
42025-02-11T20:45:01.844ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::17]:32345
52025-02-11T20:45:01.844ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
62025-02-11T20:45:01.872ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
72025-02-11T20:45:02.026ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::17]:32345
82025-02-11T20:45:20.177ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::c]:54873
92025-02-11T20:45:20.262ZINFOcrucible-agent (datafile): region 847e760a-1efe-42e7-ad14-12bd4346cb68 state: Requested
102025-02-11T20:45:20.279ZINFOcrucible-agent (dropshot): request completed latency_us = 64913 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54873 req_id = e391be6c-868d-4d88-bd38-6df51eea34f5 response_code = 200 uri = /crucible/0/regions
112025-02-11T20:45:20.279ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
122025-02-11T20:45:20.287ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_d57f35fc-ef0a-4e26-8a23-712fdd5557d6/crucible/regions/847e760a-1efe-42e7-ad14-12bd4346cb68
132025-02-11T20:45:20.287ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_d57f35fc-ef0a-4e26-8a23-712fdd5557d6/crucible/regions/847e760a-1efe-42e7-ad14-12bd4346cb68
142025-02-11T20:45:20.439ZINFOcrucible-agent (worker): creating region Region { id: RegionId("847e760a-1efe-42e7-ad14-12bd4346cb68"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/847e760a-1efe-42e7-ad14-12bd4346cb68" region = 847e760a-1efe-42e7-ad14-12bd4346cb68
152025-02-11T20:45:20.452ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "847e760a-1efe-42e7-ad14-12bd4346cb68" "--data" "/data/regions/847e760a-1efe-42e7-ad14-12bd4346cb68" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 847e760a-1efe-42e7-ad14-12bd4346cb68
162025-02-11T20:45:20.568ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54873 req_id = ae096388-4a4e-42d8-918f-583ab5fb26f6 response_code = 200 uri = /crucible/0/regions
172025-02-11T20:45:21.026ZINFOcrucible-agent (worker): region files created ok region = 847e760a-1efe-42e7-ad14-12bd4346cb68
182025-02-11T20:45:21.026ZINFOcrucible-agent (datafile): region 847e760a-1efe-42e7-ad14-12bd4346cb68 state: Requested -> Created
192025-02-11T20:45:21.026ZINFOcrucible-agent (worker): applying SMF actions post create...
202025-02-11T20:45:21.026ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-847e760a-1efe-42e7-ad14-12bd4346cb68
212025-02-11T20:45:21.039ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-847e760a-1efe-42e7-ad14-12bd4346cb68
222025-02-11T20:45:21.057ZINFOcrucible-agent (dropshot): request completed latency_us = 46 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54873 req_id = 8c88b71d-bc69-4b1c-b42f-cfce3cebeb82 response_code = 200 uri = /crucible/0/regions
232025-02-11T20:45:21.086ZINFOcrucible-agent (worker): creating config property group
242025-02-11T20:45:21.100ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-847e760a-1efe-42e7-ad14-12bd4346cb68
252025-02-11T20:45:21.101ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/847e760a-1efe-42e7-ad14-12bd4346cb68
262025-02-11T20:45:21.104ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
272025-02-11T20:45:21.104ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
282025-02-11T20:45:21.104ZINFOcrucible-agent (worker): commit
292025-02-11T20:45:21.156ZINFOcrucible-agent (worker): ok!
302025-02-11T20:45:21.177ZINFOcrucible-agent (worker): SMF ok!
312025-02-11T20:45:22.898ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:54873 req_id = 4d307ee2-ec5c-4d55-981a-68c4efb1e765 response_code = 200 uri = /crucible/0/regions/847e760a-1efe-42e7-ad14-12bd4346cb68
322025-02-11T20:45:22.901ZINFOcrucible-agent (datafile): path is "/data/regions/847e760a-1efe-42e7-ad14-12bd4346cb68"
332025-02-11T20:45:22.912ZINFOcrucible-agent (datafile): dataset is oxp_d57f35fc-ef0a-4e26-8a23-712fdd5557d6/crucible/regions/847e760a-1efe-42e7-ad14-12bd4346cb68
342025-02-11T20:45:22.922ZINFOcrucible-agent (datafile): region 847e760a-1efe-42e7-ad14-12bd4346cb68 state: Created -> Tombstoned
352025-02-11T20:45:22.922ZINFOcrucible-agent (worker): applying SMF actions before removal...
362025-02-11T20:45:22.923ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-847e760a-1efe-42e7-ad14-12bd4346cb68 (instance states: (Some(Online), None))
372025-02-11T20:45:22.930ZINFOcrucible-agent (dropshot): request completed latency_us = 28650 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:54873 req_id = 8d22ed19-55be-4b53-a856-4c7a16c6d844 response_code = 204 uri = /crucible/0/regions/847e760a-1efe-42e7-ad14-12bd4346cb68
382025-02-11T20:45:22.932ZINFOcrucible-agent (dropshot): request completed latency_us = 22 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:54873 req_id = cb187a87-36e7-4c4e-a94a-ac9b45d11bd9 response_code = 200 uri = /crucible/0/regions/847e760a-1efe-42e7-ad14-12bd4346cb68
392025-02-11T20:45:22.945ZINFOcrucible-agent (worker): SMF ok!
402025-02-11T20:45:22.957ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_d57f35fc-ef0a-4e26-8a23-712fdd5557d6/crucible/regions/847e760a-1efe-42e7-ad14-12bd4346cb68" region = 847e760a-1efe-42e7-ad14-12bd4346cb68
412025-02-11T20:45:23.019ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:54873 req_id = 0724d60b-ab30-4305-bd00-5d92f3ad5199 response_code = 200 uri = /crucible/0/regions/847e760a-1efe-42e7-ad14-12bd4346cb68
422025-02-11T20:45:23.094ZINFOcrucible-agent (datafile): region 847e760a-1efe-42e7-ad14-12bd4346cb68 state: Tombstoned -> Destroyed
432025-02-11T20:45:23.115ZINFOcrucible-agent (dropshot): request completed latency_us = 26 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:54873 req_id = 5e65f949-3080-4bda-aec2-897906543fa9 response_code = 200 uri = /crucible/0/regions/847e760a-1efe-42e7-ad14-12bd4346cb68
442025-02-11T20:55:53.094ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::c]:54872
452025-02-11T20:55:53.094ZINFOcrucible-agent (datafile): region f0b7ab7e-3053-4970-a6a5-c74940de3a5f state: Requested
462025-02-11T20:55:53.099ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
472025-02-11T20:55:53.099ZINFOcrucible-agent (dropshot): request completed latency_us = 5036 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54872 req_id = 063900bb-2c5b-4a5e-bf67-8b809f66fe51 response_code = 200 uri = /crucible/0/regions
482025-02-11T20:55:53.111ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_d57f35fc-ef0a-4e26-8a23-712fdd5557d6/crucible/regions/f0b7ab7e-3053-4970-a6a5-c74940de3a5f
492025-02-11T20:55:53.111ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_d57f35fc-ef0a-4e26-8a23-712fdd5557d6/crucible/regions/f0b7ab7e-3053-4970-a6a5-c74940de3a5f
502025-02-11T20:55:53.161ZINFOcrucible-agent (worker): creating region Region { id: RegionId("f0b7ab7e-3053-4970-a6a5-c74940de3a5f"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/f0b7ab7e-3053-4970-a6a5-c74940de3a5f" region = f0b7ab7e-3053-4970-a6a5-c74940de3a5f
512025-02-11T20:55:53.161ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "f0b7ab7e-3053-4970-a6a5-c74940de3a5f" "--data" "/data/regions/f0b7ab7e-3053-4970-a6a5-c74940de3a5f" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = f0b7ab7e-3053-4970-a6a5-c74940de3a5f
522025-02-11T20:55:53.239ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54872 req_id = 70458c06-bc76-4133-a158-cf8dd02ac538 response_code = 200 uri = /crucible/0/regions
532025-02-11T20:55:53.388ZINFOcrucible-agent (worker): region files created ok region = f0b7ab7e-3053-4970-a6a5-c74940de3a5f
542025-02-11T20:55:53.388ZINFOcrucible-agent (datafile): region f0b7ab7e-3053-4970-a6a5-c74940de3a5f state: Requested -> Created
552025-02-11T20:55:53.388ZINFOcrucible-agent (worker): applying SMF actions post create...
562025-02-11T20:55:53.390ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-f0b7ab7e-3053-4970-a6a5-c74940de3a5f
572025-02-11T20:55:53.394ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-f0b7ab7e-3053-4970-a6a5-c74940de3a5f
582025-02-11T20:55:53.407ZINFOcrucible-agent (worker): creating config property group
592025-02-11T20:55:53.410ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-f0b7ab7e-3053-4970-a6a5-c74940de3a5f
602025-02-11T20:55:53.410ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/f0b7ab7e-3053-4970-a6a5-c74940de3a5f
612025-02-11T20:55:53.410ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
622025-02-11T20:55:53.410ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
632025-02-11T20:55:53.410ZINFOcrucible-agent (worker): commit
642025-02-11T20:55:53.414ZINFOcrucible-agent (worker): ok!
652025-02-11T20:55:53.418ZINFOcrucible-agent (worker): SMF ok!
662025-02-11T20:55:53.960ZINFOcrucible-agent (dropshot): request completed latency_us = 64 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54872 req_id = 7398f8a8-f8a3-4cc3-b401-b65e4003c972 response_code = 200 uri = /crucible/0/regions