Max level shown:
1 [ Mar 25 00:15:31 Enabled. ]
2 [ Mar 25 00:15:31 Rereading configuration. ]
3 [ Mar 25 00:15:32 Rereading configuration. ]
4 [ Mar 25 00:15:34 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_4a13c90e-3ce5-4bc8-a207-b8bee3929908/crucible -l [fd00:1122:3344:101::19]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-03-25T00:15:34.904ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 7103cd3a3d7b0112d2949dd135db06fef0c156bb Commit timestamp: 2025-11-10T22:24:53.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-03-25T00:15:34.905ZINFOcrucible-agent: dataset: "oxp_4a13c90e-3ce5-4bc8-a207-b8bee3929908/crucible"
82026-03-25T00:15:34.905ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::19]:32345
92026-03-25T00:15:34.905ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-03-25T00:15:34.923ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-03-25T00:15:35.013ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::19]:32345
122026-03-25T00:16:45.506ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::c]:41799
132026-03-25T00:16:45.530ZINFOcrucible-agent (datafile): region abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3 state: Requested
142026-03-25T00:16:45.532ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152026-03-25T00:16:45.532ZINFOcrucible-agent (dropshot): request completed latency_us = 13038 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:41799 req_id = 40aa4baf-8af0-4ecd-a76f-c15277f923d2 response_code = 200 uri = /crucible/0/regions
162026-03-25T00:16:45.542ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_4a13c90e-3ce5-4bc8-a207-b8bee3929908/crucible/regions/abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
172026-03-25T00:16:45.542ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_4a13c90e-3ce5-4bc8-a207-b8bee3929908/crucible/regions/abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
182026-03-25T00:16:45.587ZINFOcrucible-agent (worker): creating region Region { id: RegionId("abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3" region = abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
192026-03-25T00:16:45.589ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3" "--data" "/data/regions/abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
202026-03-25T00:16:45.849ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:41799 req_id = b47bc505-c4f6-44da-9a67-9f5aca3e1b2f response_code = 200 uri = /crucible/0/regions
212026-03-25T00:16:45.937ZINFOcrucible-agent (worker): region files created ok region = abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
222026-03-25T00:16:45.937ZINFOcrucible-agent (datafile): region abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3 state: Requested -> Created
232026-03-25T00:16:45.937ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-03-25T00:16:45.937ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
252026-03-25T00:16:45.944ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
262026-03-25T00:16:45.961ZINFOcrucible-agent (worker): creating config property group
272026-03-25T00:16:45.964ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
282026-03-25T00:16:45.965ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
292026-03-25T00:16:45.965ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-03-25T00:16:45.965ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
312026-03-25T00:16:45.965ZINFOcrucible-agent (worker): commit
322026-03-25T00:16:45.970ZINFOcrucible-agent (worker): ok!
332026-03-25T00:16:45.974ZINFOcrucible-agent (worker): SMF ok!
342026-03-25T00:16:46.509ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:41799 req_id = f6677430-a5c6-47a6-99a5-a1b9953ee8f8 response_code = 200 uri = /crucible/0/regions
352026-03-25T00:16:46.855ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:41799 req_id = d84d30e5-228b-45aa-8621-d5d9582d0d44 response_code = 200 uri = /crucible/0/regions/abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
362026-03-25T00:16:46.860ZINFOcrucible-agent (datafile): path is "/data/regions/abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3"
372026-03-25T00:16:46.869ZINFOcrucible-agent (datafile): dataset is oxp_4a13c90e-3ce5-4bc8-a207-b8bee3929908/crucible/regions/abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
382026-03-25T00:16:46.876ZINFOcrucible-agent (datafile): region abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3 state: Created -> Tombstoned
392026-03-25T00:16:46.877ZINFOcrucible-agent (worker): applying SMF actions before removal...
402026-03-25T00:16:46.877ZINFOcrucible-agent (dropshot): request completed latency_us = 16504 local_addr = [fd00:1122:3344:101::19]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:41799 req_id = 2c0837c3-760e-492a-a368-27db81a5eaa2 response_code = 204 uri = /crucible/0/regions/abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
412026-03-25T00:16:46.878ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3 (instance states: (Some(Online), None))
422026-03-25T00:16:46.880ZINFOcrucible-agent (dropshot): request completed latency_us = 30 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:41799 req_id = 5c94949c-4b0a-4527-93ca-e6962fde6266 response_code = 200 uri = /crucible/0/regions/abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
432026-03-25T00:16:46.883ZINFOcrucible-agent (worker): SMF ok!
442026-03-25T00:16:46.891ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_4a13c90e-3ce5-4bc8-a207-b8bee3929908/crucible/regions/abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3" region = abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
452026-03-25T00:16:46.946ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:41799 req_id = 277efbc3-9714-47e8-a54e-d33dce71eed7 response_code = 200 uri = /crucible/0/regions/abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
462026-03-25T00:16:46.951ZINFOcrucible-agent (datafile): region abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3 state: Tombstoned -> Destroyed
472026-03-25T00:16:47.074ZINFOcrucible-agent (dropshot): request completed latency_us = 42 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:41799 req_id = e0648dfe-c053-4dbf-9676-4309ff1ac021 response_code = 200 uri = /crucible/0/regions/abee677b-7d6d-4dd1-bff7-1adc3b5dfbb3
482026-03-25T00:27:03.474ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::c]:46532
492026-03-25T00:27:03.475ZINFOcrucible-agent (datafile): region 8d946e55-cb41-432d-b419-b286b386bffe state: Requested
502026-03-25T00:27:03.475ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
512026-03-25T00:27:03.475ZINFOcrucible-agent (dropshot): request completed latency_us = 700 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:46532 req_id = cb7ee049-1035-428e-acf5-d18415b15dc1 response_code = 200 uri = /crucible/0/regions
522026-03-25T00:27:03.484ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_4a13c90e-3ce5-4bc8-a207-b8bee3929908/crucible/regions/8d946e55-cb41-432d-b419-b286b386bffe
532026-03-25T00:27:03.484ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_4a13c90e-3ce5-4bc8-a207-b8bee3929908/crucible/regions/8d946e55-cb41-432d-b419-b286b386bffe
542026-03-25T00:27:03.525ZINFOcrucible-agent (worker): creating region Region { id: RegionId("8d946e55-cb41-432d-b419-b286b386bffe"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/8d946e55-cb41-432d-b419-b286b386bffe" region = 8d946e55-cb41-432d-b419-b286b386bffe
552026-03-25T00:27:03.525ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "8d946e55-cb41-432d-b419-b286b386bffe" "--data" "/data/regions/8d946e55-cb41-432d-b419-b286b386bffe" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 8d946e55-cb41-432d-b419-b286b386bffe
562026-03-25T00:27:03.759ZINFOcrucible-agent (dropshot): request completed latency_us = 87 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:46532 req_id = 826192cc-fd9d-4381-a5dc-df2b7c108bf9 response_code = 200 uri = /crucible/0/regions
572026-03-25T00:27:03.838ZINFOcrucible-agent (worker): region files created ok region = 8d946e55-cb41-432d-b419-b286b386bffe
582026-03-25T00:27:03.838ZINFOcrucible-agent (datafile): region 8d946e55-cb41-432d-b419-b286b386bffe state: Requested -> Created
592026-03-25T00:27:03.838ZINFOcrucible-agent (worker): applying SMF actions post create...
602026-03-25T00:27:03.839ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-8d946e55-cb41-432d-b419-b286b386bffe
612026-03-25T00:27:03.850ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-8d946e55-cb41-432d-b419-b286b386bffe
622026-03-25T00:27:03.874ZINFOcrucible-agent (worker): creating config property group
632026-03-25T00:27:03.877ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-8d946e55-cb41-432d-b419-b286b386bffe
642026-03-25T00:27:03.878ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/8d946e55-cb41-432d-b419-b286b386bffe
652026-03-25T00:27:03.878ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
662026-03-25T00:27:03.878ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
672026-03-25T00:27:03.878ZINFOcrucible-agent (worker): commit
682026-03-25T00:27:03.882ZINFOcrucible-agent (worker): ok!
692026-03-25T00:27:03.886ZINFOcrucible-agent (worker): SMF ok!
702026-03-25T00:27:04.504ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:46532 req_id = 7e745a5e-367f-444b-bd06-48071dc33ee1 response_code = 200 uri = /crucible/0/regions
712026-03-25T00:27:10.943ZINFOcrucible-agent (datafile): region 52c360fd-9e39-4ea7-8067-0a133a4d985e state: Requested
722026-03-25T00:27:10.943ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
732026-03-25T00:27:10.943ZINFOcrucible-agent (dropshot): request completed latency_us = 237 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:46532 req_id = 3b1d2f63-eb97-4d1c-9d8b-99a22f3476e6 response_code = 200 uri = /crucible/0/regions
742026-03-25T00:27:10.949ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_4a13c90e-3ce5-4bc8-a207-b8bee3929908/crucible/regions/52c360fd-9e39-4ea7-8067-0a133a4d985e
752026-03-25T00:27:10.949ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_4a13c90e-3ce5-4bc8-a207-b8bee3929908/crucible/regions/52c360fd-9e39-4ea7-8067-0a133a4d985e
762026-03-25T00:27:11.003ZINFOcrucible-agent (worker): creating region Region { id: RegionId("52c360fd-9e39-4ea7-8067-0a133a4d985e"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/52c360fd-9e39-4ea7-8067-0a133a4d985e" region = 52c360fd-9e39-4ea7-8067-0a133a4d985e
772026-03-25T00:27:11.003ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "52c360fd-9e39-4ea7-8067-0a133a4d985e" "--data" "/data/regions/52c360fd-9e39-4ea7-8067-0a133a4d985e" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 52c360fd-9e39-4ea7-8067-0a133a4d985e
782026-03-25T00:27:11.079ZINFOcrucible-agent (worker): region files created ok region = 52c360fd-9e39-4ea7-8067-0a133a4d985e
792026-03-25T00:27:11.079ZINFOcrucible-agent (datafile): region 52c360fd-9e39-4ea7-8067-0a133a4d985e state: Requested -> Created
802026-03-25T00:27:11.079ZINFOcrucible-agent (worker): applying SMF actions post create...
812026-03-25T00:27:11.080ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-52c360fd-9e39-4ea7-8067-0a133a4d985e
822026-03-25T00:27:11.088ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-52c360fd-9e39-4ea7-8067-0a133a4d985e
832026-03-25T00:27:11.095ZINFOcrucible-agent (worker): creating config property group
842026-03-25T00:27:11.098ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-52c360fd-9e39-4ea7-8067-0a133a4d985e
852026-03-25T00:27:11.098ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/52c360fd-9e39-4ea7-8067-0a133a4d985e
862026-03-25T00:27:11.098ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
872026-03-25T00:27:11.098ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
882026-03-25T00:27:11.099ZINFOcrucible-agent (worker): commit
892026-03-25T00:27:11.102ZINFOcrucible-agent (worker): ok!
902026-03-25T00:27:11.112ZINFOcrucible-agent (worker): SMF ok!
912026-03-25T00:27:11.323ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:46532 req_id = fe5841a1-f608-4e92-9801-a51e2a33ed98 response_code = 200 uri = /crucible/0/regions
922026-03-25T00:27:36.883ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:46532 req_id = e6c92a8c-1c01-4e00-85cd-1ae4fa295868 response_code = 200 uri = /crucible/0/regions/52c360fd-9e39-4ea7-8067-0a133a4d985e
932026-03-25T00:27:36.886ZINFOcrucible-agent (datafile): path is "/data/regions/52c360fd-9e39-4ea7-8067-0a133a4d985e"
942026-03-25T00:27:36.897ZINFOcrucible-agent (datafile): dataset is oxp_4a13c90e-3ce5-4bc8-a207-b8bee3929908/crucible/regions/52c360fd-9e39-4ea7-8067-0a133a4d985e
952026-03-25T00:27:36.904ZINFOcrucible-agent (datafile): region 52c360fd-9e39-4ea7-8067-0a133a4d985e state: Created -> Tombstoned
962026-03-25T00:27:36.904ZINFOcrucible-agent (dropshot): request completed latency_us = 18526 local_addr = [fd00:1122:3344:101::19]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:46532 req_id = c5ff5408-f822-4ffe-bb50-d8228560dd1e response_code = 204 uri = /crucible/0/regions/52c360fd-9e39-4ea7-8067-0a133a4d985e
972026-03-25T00:27:36.904ZINFOcrucible-agent (worker): applying SMF actions before removal...
982026-03-25T00:27:36.906ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-52c360fd-9e39-4ea7-8067-0a133a4d985e (instance states: (Some(Online), None))
992026-03-25T00:27:36.907ZINFOcrucible-agent (dropshot): request completed latency_us = 24 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:46532 req_id = 5ad49f7b-4e6b-4bd8-a4b8-3ad3140044c9 response_code = 200 uri = /crucible/0/regions/52c360fd-9e39-4ea7-8067-0a133a4d985e
1002026-03-25T00:27:36.917ZINFOcrucible-agent (worker): SMF ok!
1012026-03-25T00:27:36.925ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_4a13c90e-3ce5-4bc8-a207-b8bee3929908/crucible/regions/52c360fd-9e39-4ea7-8067-0a133a4d985e" region = 52c360fd-9e39-4ea7-8067-0a133a4d985e
1022026-03-25T00:27:36.972ZINFOcrucible-agent (datafile): region 52c360fd-9e39-4ea7-8067-0a133a4d985e state: Tombstoned -> Destroyed
1032026-03-25T00:27:37.021ZINFOcrucible-agent (dropshot): request completed latency_us = 68 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:46532 req_id = 9dfa3634-1cc1-47e0-914e-db4a8001c17c response_code = 200 uri = /crucible/0/regions/52c360fd-9e39-4ea7-8067-0a133a4d985e