Max level shown:
1 [ Jul 23 07:46:40 Enabled. ]
2 [ Jul 23 07:46:40 Rereading configuration. ]
3 [ Jul 23 07:46:42 Rereading configuration. ]
4 [ Jul 23 07:46:46 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_2f2dde38-7c32-463a-81ec-e9938296aff2/crucible -l [fd00:1122:3344:101::17]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-07-23T07:46:47.209ZINFOcrucible-agent: dataset: "oxp_2f2dde38-7c32-463a-81ec-e9938296aff2/crucible"
72025-07-23T07:46:47.212ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::17]:32345
82025-07-23T07:46:47.212ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92025-07-23T07:46:47.229ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102025-07-23T07:46:47.348ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::17]:32345
112025-07-23T07:57:52.944ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::c]:39682
122025-07-23T07:57:52.985ZINFOcrucible-agent (datafile): region 5de29da7-aea4-47b5-8efa-c06595dc12d2 state: Requested
132025-07-23T07:57:52.990ZINFOcrucible-agent (dropshot): request completed latency_us = 23112 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:39682 req_id = a45e61bb-40d3-4b03-a4c7-69502565d93d response_code = 200 uri = /crucible/0/regions
142025-07-23T07:57:52.992ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152025-07-23T07:57:53.005ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_2f2dde38-7c32-463a-81ec-e9938296aff2/crucible/regions/5de29da7-aea4-47b5-8efa-c06595dc12d2
162025-07-23T07:57:53.005ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_2f2dde38-7c32-463a-81ec-e9938296aff2/crucible/regions/5de29da7-aea4-47b5-8efa-c06595dc12d2
172025-07-23T07:57:53.064ZINFOcrucible-agent (worker): creating region Region { id: RegionId("5de29da7-aea4-47b5-8efa-c06595dc12d2"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/5de29da7-aea4-47b5-8efa-c06595dc12d2" region = 5de29da7-aea4-47b5-8efa-c06595dc12d2
182025-07-23T07:57:53.066ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "5de29da7-aea4-47b5-8efa-c06595dc12d2" "--data" "/data/regions/5de29da7-aea4-47b5-8efa-c06595dc12d2" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 5de29da7-aea4-47b5-8efa-c06595dc12d2
192025-07-23T07:57:53.239ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:39682 req_id = c2273ef2-9edf-4c71-b380-f2361bab537e response_code = 200 uri = /crucible/0/regions
202025-07-23T07:57:53.386ZINFOcrucible-agent (worker): region files created ok region = 5de29da7-aea4-47b5-8efa-c06595dc12d2
212025-07-23T07:57:53.386ZINFOcrucible-agent (datafile): region 5de29da7-aea4-47b5-8efa-c06595dc12d2 state: Requested -> Created
222025-07-23T07:57:53.386ZINFOcrucible-agent (worker): applying SMF actions post create...
232025-07-23T07:57:53.387ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-5de29da7-aea4-47b5-8efa-c06595dc12d2
242025-07-23T07:57:53.397ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-5de29da7-aea4-47b5-8efa-c06595dc12d2
252025-07-23T07:57:53.419ZINFOcrucible-agent (worker): creating config property group
262025-07-23T07:57:53.422ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5de29da7-aea4-47b5-8efa-c06595dc12d2
272025-07-23T07:57:53.422ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5de29da7-aea4-47b5-8efa-c06595dc12d2
282025-07-23T07:57:53.422ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292025-07-23T07:57:53.422ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
302025-07-23T07:57:53.422ZINFOcrucible-agent (worker): commit
312025-07-23T07:57:53.426ZINFOcrucible-agent (worker): ok!
322025-07-23T07:57:53.444ZINFOcrucible-agent (worker): SMF ok!
332025-07-23T07:57:53.894ZINFOcrucible-agent (dropshot): request completed latency_us = 50 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:39682 req_id = 7da09c2a-b919-4037-a16b-8f32cef5513e response_code = 200 uri = /crucible/0/regions
342025-07-23T07:57:59.535ZINFOcrucible-agent (datafile): region bedd1856-dbff-46c0-84b9-9bab179104f3 state: Requested
352025-07-23T07:57:59.535ZINFOcrucible-agent (dropshot): request completed latency_us = 240 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:39682 req_id = 7fb63d5a-95cd-4d73-80a6-22212a604bb5 response_code = 200 uri = /crucible/0/regions
362025-07-23T07:57:59.535ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
372025-07-23T07:57:59.544ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_2f2dde38-7c32-463a-81ec-e9938296aff2/crucible/regions/bedd1856-dbff-46c0-84b9-9bab179104f3
382025-07-23T07:57:59.544ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_2f2dde38-7c32-463a-81ec-e9938296aff2/crucible/regions/bedd1856-dbff-46c0-84b9-9bab179104f3
392025-07-23T07:57:59.604ZINFOcrucible-agent (worker): creating region Region { id: RegionId("bedd1856-dbff-46c0-84b9-9bab179104f3"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/bedd1856-dbff-46c0-84b9-9bab179104f3" region = bedd1856-dbff-46c0-84b9-9bab179104f3
402025-07-23T07:57:59.604ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "bedd1856-dbff-46c0-84b9-9bab179104f3" "--data" "/data/regions/bedd1856-dbff-46c0-84b9-9bab179104f3" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = bedd1856-dbff-46c0-84b9-9bab179104f3
412025-07-23T07:57:59.687ZINFOcrucible-agent (worker): region files created ok region = bedd1856-dbff-46c0-84b9-9bab179104f3
422025-07-23T07:57:59.687ZINFOcrucible-agent (datafile): region bedd1856-dbff-46c0-84b9-9bab179104f3 state: Requested -> Created
432025-07-23T07:57:59.687ZINFOcrucible-agent (worker): applying SMF actions post create...
442025-07-23T07:57:59.689ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-bedd1856-dbff-46c0-84b9-9bab179104f3
452025-07-23T07:57:59.692ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-bedd1856-dbff-46c0-84b9-9bab179104f3
462025-07-23T07:57:59.699ZINFOcrucible-agent (worker): creating config property group
472025-07-23T07:57:59.702ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-bedd1856-dbff-46c0-84b9-9bab179104f3
482025-07-23T07:57:59.703ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/bedd1856-dbff-46c0-84b9-9bab179104f3
492025-07-23T07:57:59.703ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
502025-07-23T07:57:59.703ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
512025-07-23T07:57:59.703ZINFOcrucible-agent (worker): commit
522025-07-23T07:57:59.707ZINFOcrucible-agent (worker): ok!
532025-07-23T07:57:59.711ZINFOcrucible-agent (worker): SMF ok!
542025-07-23T07:57:59.743ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:39682 req_id = 76ce9d91-5570-461d-b909-0081cb9524c9 response_code = 200 uri = /crucible/0/regions
552025-07-23T07:58:24.521ZINFOcrucible-agent (dropshot): request completed latency_us = 89 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:39682 req_id = 940b4066-4a58-4876-aeec-79b386cde3f3 response_code = 200 uri = /crucible/0/regions/bedd1856-dbff-46c0-84b9-9bab179104f3
562025-07-23T07:58:24.525ZINFOcrucible-agent (datafile): path is "/data/regions/bedd1856-dbff-46c0-84b9-9bab179104f3"
572025-07-23T07:58:24.537ZINFOcrucible-agent (datafile): dataset is oxp_2f2dde38-7c32-463a-81ec-e9938296aff2/crucible/regions/bedd1856-dbff-46c0-84b9-9bab179104f3
582025-07-23T07:58:24.547ZINFOcrucible-agent (datafile): region bedd1856-dbff-46c0-84b9-9bab179104f3 state: Created -> Tombstoned
592025-07-23T07:58:24.548ZINFOcrucible-agent (worker): applying SMF actions before removal...
602025-07-23T07:58:24.548ZINFOcrucible-agent (dropshot): request completed latency_us = 22524 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:39682 req_id = e25073a8-2c1b-4f87-b5ee-1f9d7e0fdd31 response_code = 204 uri = /crucible/0/regions/bedd1856-dbff-46c0-84b9-9bab179104f3
612025-07-23T07:58:24.549ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-bedd1856-dbff-46c0-84b9-9bab179104f3 (instance states: (Some(Online), None))
622025-07-23T07:58:24.551ZINFOcrucible-agent (dropshot): request completed latency_us = 37 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:39682 req_id = 0a0d6e05-2001-4e99-861a-fd328a58811d response_code = 200 uri = /crucible/0/regions/bedd1856-dbff-46c0-84b9-9bab179104f3
632025-07-23T07:58:24.557ZINFOcrucible-agent (worker): SMF ok!
642025-07-23T07:58:24.570ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_2f2dde38-7c32-463a-81ec-e9938296aff2/crucible/regions/bedd1856-dbff-46c0-84b9-9bab179104f3" region = bedd1856-dbff-46c0-84b9-9bab179104f3
652025-07-23T07:58:24.609ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:39682 req_id = 4250311d-14a4-4a44-a066-09f2544b24e4 response_code = 200 uri = /crucible/0/regions/bedd1856-dbff-46c0-84b9-9bab179104f3
662025-07-23T07:58:24.636ZINFOcrucible-agent (datafile): region bedd1856-dbff-46c0-84b9-9bab179104f3 state: Tombstoned -> Destroyed
672025-07-23T07:58:24.780ZINFOcrucible-agent (dropshot): request completed latency_us = 33 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:39682 req_id = d12960e4-85f1-4c34-8578-aac0f2f85a2e response_code = 200 uri = /crucible/0/regions/bedd1856-dbff-46c0-84b9-9bab179104f3