Max level shown:
1 [ Dec 18 18:30:41 Enabled. ]
2 [ Dec 18 18:30:41 Rereading configuration. ]
3 [ Dec 18 18:30:42 Rereading configuration. ]
4 [ Dec 18 18:30:46 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_68a261bd-5c4f-492b-8483-b232ebc5f6af/crucible -l [fd00:1122:3344:101::1a]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-12-18T18:30:46.948ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 7103cd3a3d7b0112d2949dd135db06fef0c156bb Commit timestamp: 2025-11-10T22:24:53.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72025-12-18T18:30:46.949ZINFOcrucible-agent: dataset: "oxp_68a261bd-5c4f-492b-8483-b232ebc5f6af/crucible"
82025-12-18T18:30:46.949ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::1a]:32345
92025-12-18T18:30:46.949ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102025-12-18T18:30:46.963ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112025-12-18T18:30:47.063ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::1a]:32345
122025-12-18T18:31:45.673ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::b]:45056
132025-12-18T18:31:45.708ZINFOcrucible-agent (datafile): region 869f6edc-9b9c-48bf-84bb-c1f5ce3161ab state: Requested
142025-12-18T18:31:45.711ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152025-12-18T18:31:45.711ZINFOcrucible-agent (dropshot): request completed latency_us = 16444 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:45056 req_id = b6b4c495-e7cf-446f-bba3-69f8ecc69093 response_code = 200 uri = /crucible/0/regions
162025-12-18T18:31:45.720ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_68a261bd-5c4f-492b-8483-b232ebc5f6af/crucible/regions/869f6edc-9b9c-48bf-84bb-c1f5ce3161ab
172025-12-18T18:31:45.720ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_68a261bd-5c4f-492b-8483-b232ebc5f6af/crucible/regions/869f6edc-9b9c-48bf-84bb-c1f5ce3161ab
182025-12-18T18:31:45.781ZINFOcrucible-agent (worker): creating region Region { id: RegionId("869f6edc-9b9c-48bf-84bb-c1f5ce3161ab"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/869f6edc-9b9c-48bf-84bb-c1f5ce3161ab" region = 869f6edc-9b9c-48bf-84bb-c1f5ce3161ab
192025-12-18T18:31:45.783ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "869f6edc-9b9c-48bf-84bb-c1f5ce3161ab" "--data" "/data/regions/869f6edc-9b9c-48bf-84bb-c1f5ce3161ab" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 869f6edc-9b9c-48bf-84bb-c1f5ce3161ab
202025-12-18T18:31:45.870ZINFOcrucible-agent (dropshot): request completed latency_us = 72 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:45056 req_id = 30481a9c-5467-421e-a05f-a0616b80bc2d response_code = 200 uri = /crucible/0/regions
212025-12-18T18:31:46.142ZINFOcrucible-agent (worker): region files created ok region = 869f6edc-9b9c-48bf-84bb-c1f5ce3161ab
222025-12-18T18:31:46.142ZINFOcrucible-agent (datafile): region 869f6edc-9b9c-48bf-84bb-c1f5ce3161ab state: Requested -> Created
232025-12-18T18:31:46.142ZINFOcrucible-agent (worker): applying SMF actions post create...
242025-12-18T18:31:46.143ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-869f6edc-9b9c-48bf-84bb-c1f5ce3161ab
252025-12-18T18:31:46.150ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-869f6edc-9b9c-48bf-84bb-c1f5ce3161ab
262025-12-18T18:31:46.168ZINFOcrucible-agent (worker): creating config property group
272025-12-18T18:31:46.174ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-869f6edc-9b9c-48bf-84bb-c1f5ce3161ab
282025-12-18T18:31:46.174ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/869f6edc-9b9c-48bf-84bb-c1f5ce3161ab
292025-12-18T18:31:46.175ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302025-12-18T18:31:46.175ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
312025-12-18T18:31:46.175ZINFOcrucible-agent (worker): commit
322025-12-18T18:31:46.181ZINFOcrucible-agent (worker): ok!
332025-12-18T18:31:46.188ZINFOcrucible-agent (worker): SMF ok!
342025-12-18T18:31:46.254ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:45056 req_id = 14967fd0-e975-4a2f-806a-ab8d3c3616b2 response_code = 200 uri = /crucible/0/regions
352025-12-18T18:31:47.198ZINFOcrucible-agent (dropshot): request completed latency_us = 82 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:45056 req_id = c945dd95-5650-40b6-959d-8d9021cf8a47 response_code = 200 uri = /crucible/0/regions/869f6edc-9b9c-48bf-84bb-c1f5ce3161ab
362025-12-18T18:31:47.201ZINFOcrucible-agent (datafile): path is "/data/regions/869f6edc-9b9c-48bf-84bb-c1f5ce3161ab"
372025-12-18T18:31:47.209ZINFOcrucible-agent (datafile): dataset is oxp_68a261bd-5c4f-492b-8483-b232ebc5f6af/crucible/regions/869f6edc-9b9c-48bf-84bb-c1f5ce3161ab
382025-12-18T18:31:47.217ZINFOcrucible-agent (datafile): region 869f6edc-9b9c-48bf-84bb-c1f5ce3161ab state: Created -> Tombstoned
392025-12-18T18:31:47.217ZINFOcrucible-agent (worker): applying SMF actions before removal...
402025-12-18T18:31:47.217ZINFOcrucible-agent (dropshot): request completed latency_us = 16138 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:45056 req_id = 33fb5785-a0bc-43e1-af70-d9504fb039d7 response_code = 204 uri = /crucible/0/regions/869f6edc-9b9c-48bf-84bb-c1f5ce3161ab
412025-12-18T18:31:47.218ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-869f6edc-9b9c-48bf-84bb-c1f5ce3161ab (instance states: (Some(Online), None))
422025-12-18T18:31:47.220ZINFOcrucible-agent (dropshot): request completed latency_us = 31 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:45056 req_id = 04f2e548-d142-4296-97ee-e86c9dcf185a response_code = 200 uri = /crucible/0/regions/869f6edc-9b9c-48bf-84bb-c1f5ce3161ab
432025-12-18T18:31:47.222ZINFOcrucible-agent (worker): SMF ok!
442025-12-18T18:31:47.231ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_68a261bd-5c4f-492b-8483-b232ebc5f6af/crucible/regions/869f6edc-9b9c-48bf-84bb-c1f5ce3161ab" region = 869f6edc-9b9c-48bf-84bb-c1f5ce3161ab
452025-12-18T18:31:47.304ZINFOcrucible-agent (datafile): region 869f6edc-9b9c-48bf-84bb-c1f5ce3161ab state: Tombstoned -> Destroyed
462025-12-18T18:31:47.351ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:45056 req_id = bf0a70d9-7fbf-4ad8-a455-fd58c5f4efc3 response_code = 200 uri = /crucible/0/regions/869f6edc-9b9c-48bf-84bb-c1f5ce3161ab
472025-12-18T18:43:13.349ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::b]:33890
482025-12-18T18:43:13.349ZINFOcrucible-agent (datafile): region 7f3ad13b-e489-46c0-a465-49d9276d9392 state: Requested
492025-12-18T18:43:13.349ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
502025-12-18T18:43:13.349ZINFOcrucible-agent (dropshot): request completed latency_us = 596 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:33890 req_id = 8c820e6e-5b60-4eac-87d7-81d0ee3653ee response_code = 200 uri = /crucible/0/regions
512025-12-18T18:43:13.358ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_68a261bd-5c4f-492b-8483-b232ebc5f6af/crucible/regions/7f3ad13b-e489-46c0-a465-49d9276d9392
522025-12-18T18:43:13.358ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_68a261bd-5c4f-492b-8483-b232ebc5f6af/crucible/regions/7f3ad13b-e489-46c0-a465-49d9276d9392
532025-12-18T18:43:13.410ZINFOcrucible-agent (worker): creating region Region { id: RegionId("7f3ad13b-e489-46c0-a465-49d9276d9392"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/7f3ad13b-e489-46c0-a465-49d9276d9392" region = 7f3ad13b-e489-46c0-a465-49d9276d9392
542025-12-18T18:43:13.410ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "7f3ad13b-e489-46c0-a465-49d9276d9392" "--data" "/data/regions/7f3ad13b-e489-46c0-a465-49d9276d9392" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 7f3ad13b-e489-46c0-a465-49d9276d9392
552025-12-18T18:43:13.679ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:33890 req_id = 44fb89e2-4c33-45fb-86b5-9668ca1430d1 response_code = 200 uri = /crucible/0/regions
562025-12-18T18:43:13.711ZINFOcrucible-agent (worker): region files created ok region = 7f3ad13b-e489-46c0-a465-49d9276d9392
572025-12-18T18:43:13.711ZINFOcrucible-agent (datafile): region 7f3ad13b-e489-46c0-a465-49d9276d9392 state: Requested -> Created
582025-12-18T18:43:13.711ZINFOcrucible-agent (worker): applying SMF actions post create...
592025-12-18T18:43:13.712ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-7f3ad13b-e489-46c0-a465-49d9276d9392
602025-12-18T18:43:13.724ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-7f3ad13b-e489-46c0-a465-49d9276d9392
612025-12-18T18:43:13.750ZINFOcrucible-agent (worker): creating config property group
622025-12-18T18:43:13.754ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-7f3ad13b-e489-46c0-a465-49d9276d9392
632025-12-18T18:43:13.755ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/7f3ad13b-e489-46c0-a465-49d9276d9392
642025-12-18T18:43:13.755ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
652025-12-18T18:43:13.755ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
662025-12-18T18:43:13.755ZINFOcrucible-agent (worker): commit
672025-12-18T18:43:13.760ZINFOcrucible-agent (worker): ok!
682025-12-18T18:43:13.764ZINFOcrucible-agent (worker): SMF ok!
692025-12-18T18:43:14.291ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:33890 req_id = 67bc1301-5b15-4647-9240-cdd762a7ee5c response_code = 200 uri = /crucible/0/regions
702025-12-18T18:43:19.932ZINFOcrucible-agent (datafile): region bf2fb146-cdcd-418b-9634-33130c106189 state: Requested
712025-12-18T18:43:19.933ZINFOcrucible-agent (dropshot): request completed latency_us = 235 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:33890 req_id = 7a8bee10-919e-4f23-9f6e-8493b79a0ab9 response_code = 200 uri = /crucible/0/regions
722025-12-18T18:43:19.933ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
732025-12-18T18:43:19.940ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_68a261bd-5c4f-492b-8483-b232ebc5f6af/crucible/regions/bf2fb146-cdcd-418b-9634-33130c106189
742025-12-18T18:43:19.940ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_68a261bd-5c4f-492b-8483-b232ebc5f6af/crucible/regions/bf2fb146-cdcd-418b-9634-33130c106189
752025-12-18T18:43:19.994ZINFOcrucible-agent (worker): creating region Region { id: RegionId("bf2fb146-cdcd-418b-9634-33130c106189"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/bf2fb146-cdcd-418b-9634-33130c106189" region = bf2fb146-cdcd-418b-9634-33130c106189
762025-12-18T18:43:19.994ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "bf2fb146-cdcd-418b-9634-33130c106189" "--data" "/data/regions/bf2fb146-cdcd-418b-9634-33130c106189" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = bf2fb146-cdcd-418b-9634-33130c106189
772025-12-18T18:43:20.062ZINFOcrucible-agent (dropshot): request completed latency_us = 49 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:33890 req_id = fae67275-913c-4137-9fd9-517b69e9c24b response_code = 200 uri = /crucible/0/regions
782025-12-18T18:43:20.066ZINFOcrucible-agent (worker): region files created ok region = bf2fb146-cdcd-418b-9634-33130c106189
792025-12-18T18:43:20.066ZINFOcrucible-agent (datafile): region bf2fb146-cdcd-418b-9634-33130c106189 state: Requested -> Created
802025-12-18T18:43:20.066ZINFOcrucible-agent (worker): applying SMF actions post create...
812025-12-18T18:43:20.071ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-bf2fb146-cdcd-418b-9634-33130c106189
822025-12-18T18:43:20.079ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-bf2fb146-cdcd-418b-9634-33130c106189
832025-12-18T18:43:20.088ZINFOcrucible-agent (worker): creating config property group
842025-12-18T18:43:20.092ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-bf2fb146-cdcd-418b-9634-33130c106189
852025-12-18T18:43:20.093ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/bf2fb146-cdcd-418b-9634-33130c106189
862025-12-18T18:43:20.093ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
872025-12-18T18:43:20.093ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
882025-12-18T18:43:20.093ZINFOcrucible-agent (worker): commit
892025-12-18T18:43:20.096ZINFOcrucible-agent (worker): ok!
902025-12-18T18:43:20.101ZINFOcrucible-agent (worker): SMF ok!
912025-12-18T18:43:20.544ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:33890 req_id = 5cd1540e-80ff-4fb5-8566-3739d32fb7ec response_code = 200 uri = /crucible/0/regions
922025-12-18T18:43:52.145ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::b]:51417
932025-12-18T18:43:52.145ZINFOcrucible-agent (dropshot): request completed latency_us = 75 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:51417 req_id = 6be8262a-3638-4223-8e07-3377f014127c response_code = 200 uri = /crucible/0/regions/bf2fb146-cdcd-418b-9634-33130c106189
942025-12-18T18:43:52.149ZINFOcrucible-agent (datafile): path is "/data/regions/bf2fb146-cdcd-418b-9634-33130c106189"
952025-12-18T18:43:52.159ZINFOcrucible-agent (datafile): dataset is oxp_68a261bd-5c4f-492b-8483-b232ebc5f6af/crucible/regions/bf2fb146-cdcd-418b-9634-33130c106189
962025-12-18T18:43:52.167ZINFOcrucible-agent (datafile): region bf2fb146-cdcd-418b-9634-33130c106189 state: Created -> Tombstoned
972025-12-18T18:43:52.167ZINFOcrucible-agent (worker): applying SMF actions before removal...
982025-12-18T18:43:52.167ZINFOcrucible-agent (dropshot): request completed latency_us = 18245 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:51417 req_id = 5d344926-54eb-4b22-ae4f-443ab16c7ae3 response_code = 204 uri = /crucible/0/regions/bf2fb146-cdcd-418b-9634-33130c106189
992025-12-18T18:43:52.169ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-bf2fb146-cdcd-418b-9634-33130c106189 (instance states: (Some(Online), None))
1002025-12-18T18:43:52.171ZINFOcrucible-agent (dropshot): request completed latency_us = 39 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:51417 req_id = e54f9105-5820-4b5e-931f-08a3f3154ee1 response_code = 200 uri = /crucible/0/regions/bf2fb146-cdcd-418b-9634-33130c106189
1012025-12-18T18:43:52.187ZINFOcrucible-agent (worker): SMF ok!
1022025-12-18T18:43:52.195ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_68a261bd-5c4f-492b-8483-b232ebc5f6af/crucible/regions/bf2fb146-cdcd-418b-9634-33130c106189" region = bf2fb146-cdcd-418b-9634-33130c106189
1032025-12-18T18:43:52.258ZINFOcrucible-agent (datafile): region bf2fb146-cdcd-418b-9634-33130c106189 state: Tombstoned -> Destroyed
1042025-12-18T18:43:52.299ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:51417 req_id = be3c8e40-fd1a-4e42-8eeb-76ededb35fc8 response_code = 200 uri = /crucible/0/regions/bf2fb146-cdcd-418b-9634-33130c106189