Max level shown:
1 [ Feb 28 10:02:23 Enabled. ]
2 [ Feb 28 10:02:23 Rereading configuration. ]
3 [ Feb 28 10:02:25 Rereading configuration. ]
4 [ Feb 28 10:02:33 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_cad7c44e-69e1-48a0-a08c-982b51e87095/crucible -l [fd00:1122:3344:101::16]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-02-28T10:02:34.351ZINFOcrucible-agent: dataset: "oxp_cad7c44e-69e1-48a0-a08c-982b51e87095/crucible"
72025-02-28T10:02:34.354ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::16]:32345
82025-02-28T10:02:34.354ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92025-02-28T10:02:34.374ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102025-02-28T10:02:34.625ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::16]:32345
112025-02-28T10:02:52.604ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:51386
122025-02-28T10:02:52.630ZINFOcrucible-agent (datafile): region 9985ec8d-b1ef-408f-a28c-0fe7b84ff80e state: Requested
132025-02-28T10:02:52.633ZINFOcrucible-agent (dropshot): request completed latency_us = 11566 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51386 req_id = 15e1c6cc-d3e7-4bad-a5a3-b3a3800e0cea response_code = 200 uri = /crucible/0/regions
142025-02-28T10:02:52.633ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152025-02-28T10:02:52.643ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_cad7c44e-69e1-48a0-a08c-982b51e87095/crucible/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e
162025-02-28T10:02:52.643ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_cad7c44e-69e1-48a0-a08c-982b51e87095/crucible/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e
172025-02-28T10:02:52.719ZINFOcrucible-agent (worker): creating region Region { id: RegionId("9985ec8d-b1ef-408f-a28c-0fe7b84ff80e"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e" region = 9985ec8d-b1ef-408f-a28c-0fe7b84ff80e
182025-02-28T10:02:52.721ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "9985ec8d-b1ef-408f-a28c-0fe7b84ff80e" "--data" "/data/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 9985ec8d-b1ef-408f-a28c-0fe7b84ff80e
192025-02-28T10:02:52.787ZINFOcrucible-agent (dropshot): request completed latency_us = 93 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51386 req_id = 39aadfa1-b759-4e81-a025-efd8fde05468 response_code = 200 uri = /crucible/0/regions
202025-02-28T10:02:52.867ZINFOcrucible-agent (worker): region files created ok region = 9985ec8d-b1ef-408f-a28c-0fe7b84ff80e
212025-02-28T10:02:52.867ZINFOcrucible-agent (datafile): region 9985ec8d-b1ef-408f-a28c-0fe7b84ff80e state: Requested -> Created
222025-02-28T10:02:52.867ZINFOcrucible-agent (worker): applying SMF actions post create...
232025-02-28T10:02:52.868ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-9985ec8d-b1ef-408f-a28c-0fe7b84ff80e
242025-02-28T10:02:52.876ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-9985ec8d-b1ef-408f-a28c-0fe7b84ff80e
252025-02-28T10:02:52.888ZINFOcrucible-agent (worker): creating config property group
262025-02-28T10:02:52.892ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-9985ec8d-b1ef-408f-a28c-0fe7b84ff80e
272025-02-28T10:02:52.892ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e
282025-02-28T10:02:52.892ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292025-02-28T10:02:52.892ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
302025-02-28T10:02:52.892ZINFOcrucible-agent (worker): commit
312025-02-28T10:02:52.897ZINFOcrucible-agent (worker): ok!
322025-02-28T10:02:52.908ZINFOcrucible-agent (worker): SMF ok!
332025-02-28T10:02:53.512ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51386 req_id = c6e5c07f-2da6-43c2-b62d-77e07bc4124b response_code = 200 uri = /crucible/0/regions
342025-02-28T10:13:31.989ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:39128
352025-02-28T10:13:31.992ZINFOcrucible-agent (dropshot): request completed latency_us = 2937 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39128 req_id = b9899258-9153-4169-b87a-9f0017e6eb95 response_code = 200 uri = /crucible/0/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e
362025-02-28T10:13:31.998ZINFOcrucible-agent (datafile): path is "/data/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e"
372025-02-28T10:13:32.009ZINFOcrucible-agent (datafile): dataset is oxp_cad7c44e-69e1-48a0-a08c-982b51e87095/crucible/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e
382025-02-28T10:13:32.018ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_cad7c44e-69e1-48a0-a08c-982b51e87095/crucible/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e@7530d1d3-6aab-42d0-b33d-cab3169b20bf
392025-02-28T10:13:32.018ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_cad7c44e-69e1-48a0-a08c-982b51e87095/crucible/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e", "7530d1d3-6aab-42d0-b33d-cab3169b20bf"]
402025-02-28T10:13:32.026ZINFOcrucible-agent (dropshot): request completed latency_us = 30929 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39128 req_id = 79363da6-8190-4e95-9bc1-f77ffb0f94a5 response_code = 200 uri = /crucible/0/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e/snapshots/7530d1d3-6aab-42d0-b33d-cab3169b20bf
412025-02-28T10:13:32.031ZINFOcrucible-agent (datafile): path is "/data/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e"
422025-02-28T10:13:32.038ZINFOcrucible-agent (datafile): dataset is oxp_cad7c44e-69e1-48a0-a08c-982b51e87095/crucible/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e
432025-02-28T10:13:32.046ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_cad7c44e-69e1-48a0-a08c-982b51e87095/crucible/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e@7530d1d3-6aab-42d0-b33d-cab3169b20bf
442025-02-28T10:13:32.046ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_cad7c44e-69e1-48a0-a08c-982b51e87095/crucible/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e", "7530d1d3-6aab-42d0-b33d-cab3169b20bf"]
452025-02-28T10:13:32.055ZINFOcrucible-agent (datafile): requesting running snapshot 9985ec8d-b1ef-408f-a28c-0fe7b84ff80e-7530d1d3-6aab-42d0-b33d-cab3169b20bf state: Requested
462025-02-28T10:13:32.056ZINFOcrucible-agent (worker): applying SMF actions for region 9985ec8d-b1ef-408f-a28c-0fe7b84ff80e running snapshot 7530d1d3-6aab-42d0-b33d-cab3169b20bf (state Requested)...
472025-02-28T10:13:32.056ZINFOcrucible-agent (dropshot): request completed latency_us = 25746 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39128 req_id = 3f252032-32a8-43d2-be0e-483277164111 response_code = 200 uri = /crucible/0/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e/snapshots/7530d1d3-6aab-42d0-b33d-cab3169b20bf/run
482025-02-28T10:13:32.060ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-9985ec8d-b1ef-408f-a28c-0fe7b84ff80e-7530d1d3-6aab-42d0-b33d-cab3169b20bf
492025-02-28T10:13:32.064ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-9985ec8d-b1ef-408f-a28c-0fe7b84ff80e-7530d1d3-6aab-42d0-b33d-cab3169b20bf
502025-02-28T10:13:32.074ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-9985ec8d-b1ef-408f-a28c-0fe7b84ff80e-7530d1d3-6aab-42d0-b33d-cab3169b20bf
512025-02-28T10:13:32.075ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e/.zfs/snapshot/7530d1d3-6aab-42d0-b33d-cab3169b20bf
522025-02-28T10:13:32.075ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
532025-02-28T10:13:32.075ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
542025-02-28T10:13:32.075ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
552025-02-28T10:13:32.075ZINFOcrucible-agent (worker): commit
562025-02-28T10:13:32.079ZINFOcrucible-agent (worker): ok!
572025-02-28T10:13:32.083ZINFOcrucible-agent (worker): SMF ok!
582025-02-28T10:13:32.083ZINFOcrucible-agent (datafile): region 9985ec8d-b1ef-408f-a28c-0fe7b84ff80e running snapshot 7530d1d3-6aab-42d0-b33d-cab3169b20bf state: Requested -> Created
592025-02-28T10:13:32.353ZINFOcrucible-agent (datafile): path is "/data/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e"
602025-02-28T10:13:32.362ZINFOcrucible-agent (datafile): dataset is oxp_cad7c44e-69e1-48a0-a08c-982b51e87095/crucible/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e
612025-02-28T10:13:32.370ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_cad7c44e-69e1-48a0-a08c-982b51e87095/crucible/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e@7530d1d3-6aab-42d0-b33d-cab3169b20bf
622025-02-28T10:13:32.370ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_cad7c44e-69e1-48a0-a08c-982b51e87095/crucible/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e", "7530d1d3-6aab-42d0-b33d-cab3169b20bf"]
632025-02-28T10:13:32.379ZINFOcrucible-agent (dropshot): request completed latency_us = 26100 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39128 req_id = 856d6cd6-da7a-43a5-a3b8-7fb943383482 response_code = 200 uri = /crucible/0/regions/9985ec8d-b1ef-408f-a28c-0fe7b84ff80e/snapshots/7530d1d3-6aab-42d0-b33d-cab3169b20bf/run