Max level shown:
1 [ May 27 21:01:38 Enabled. ]
2 [ May 27 21:01:38 Rereading configuration. ]
3 [ May 27 21:01:39 Rereading configuration. ]
4 [ May 27 21:01:43 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_a944f5b5-20fa-47fb-9f8a-59ed228bb8db/crucible -l [fd00:1122:3344:101::14]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-05-27T21:01:43.319ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: bd9a0e2abe6b6b89aec8c85f4ee57474144ed150 Commit timestamp: 2026-05-14T15:48:00.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-05-27T21:01:43.319ZINFOcrucible-agent: dataset: "oxp_a944f5b5-20fa-47fb-9f8a-59ed228bb8db/crucible"
82026-05-27T21:01:43.319ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::14]:32345
92026-05-27T21:01:43.319ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-05-27T21:01:43.340ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-05-27T21:01:43.417ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::14]:32345
122026-05-27T21:13:14.362ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:51453
132026-05-27T21:13:14.413ZINFOcrucible-agent (datafile): region 86e12674-1371-4728-9221-3a0fb9722ae1 state: Requested
142026-05-27T21:13:14.419ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152026-05-27T21:13:14.419ZINFOcrucible-agent (dropshot): request completed latency_us = 19845 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51453 req_id = 8d74c832-fab6-4e42-8a12-e218db7436d9 response_code = 200 uri = /crucible/0/regions
162026-05-27T21:13:14.431ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_a944f5b5-20fa-47fb-9f8a-59ed228bb8db/crucible/regions/86e12674-1371-4728-9221-3a0fb9722ae1
172026-05-27T21:13:14.431ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_a944f5b5-20fa-47fb-9f8a-59ed228bb8db/crucible/regions/86e12674-1371-4728-9221-3a0fb9722ae1
182026-05-27T21:13:14.492ZINFOcrucible-agent (worker): creating region Region { id: RegionId("86e12674-1371-4728-9221-3a0fb9722ae1"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/86e12674-1371-4728-9221-3a0fb9722ae1" region = 86e12674-1371-4728-9221-3a0fb9722ae1
192026-05-27T21:13:14.494ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "86e12674-1371-4728-9221-3a0fb9722ae1" "--data" "/data/regions/86e12674-1371-4728-9221-3a0fb9722ae1" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 86e12674-1371-4728-9221-3a0fb9722ae1
202026-05-27T21:13:14.707ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51453 req_id = 6268e638-fc4f-4464-968c-da26b475fa09 response_code = 200 uri = /crucible/0/regions
212026-05-27T21:13:14.844ZINFOcrucible-agent (worker): region files created ok region = 86e12674-1371-4728-9221-3a0fb9722ae1
222026-05-27T21:13:14.844ZINFOcrucible-agent (datafile): region 86e12674-1371-4728-9221-3a0fb9722ae1 state: Requested -> Created
232026-05-27T21:13:14.845ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-05-27T21:13:14.847ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-86e12674-1371-4728-9221-3a0fb9722ae1
252026-05-27T21:13:14.857ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-86e12674-1371-4728-9221-3a0fb9722ae1
262026-05-27T21:13:14.874ZINFOcrucible-agent (worker): creating config property group
272026-05-27T21:13:14.876ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-86e12674-1371-4728-9221-3a0fb9722ae1
282026-05-27T21:13:14.876ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/86e12674-1371-4728-9221-3a0fb9722ae1
292026-05-27T21:13:14.877ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-05-27T21:13:14.877ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
312026-05-27T21:13:14.877ZINFOcrucible-agent (worker): commit
322026-05-27T21:13:14.890ZINFOcrucible-agent (worker): ok!
332026-05-27T21:13:14.892ZINFOcrucible-agent (worker): SMF ok!
342026-05-27T21:13:15.435ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51453 req_id = 1ae888fb-ef5b-4367-a04e-b9ddd484d137 response_code = 200 uri = /crucible/0/regions
352026-05-27T21:13:20.907ZINFOcrucible-agent (datafile): region 59ffb213-5b7f-4b7e-9695-c8125cdcffae state: Requested
362026-05-27T21:13:20.908ZINFOcrucible-agent (dropshot): request completed latency_us = 275 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51453 req_id = ba3c4e37-ed81-405b-a8cf-301c203cf090 response_code = 200 uri = /crucible/0/regions
372026-05-27T21:13:20.908ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
382026-05-27T21:13:20.915ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_a944f5b5-20fa-47fb-9f8a-59ed228bb8db/crucible/regions/59ffb213-5b7f-4b7e-9695-c8125cdcffae
392026-05-27T21:13:20.915ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_a944f5b5-20fa-47fb-9f8a-59ed228bb8db/crucible/regions/59ffb213-5b7f-4b7e-9695-c8125cdcffae
402026-05-27T21:13:20.963ZINFOcrucible-agent (worker): creating region Region { id: RegionId("59ffb213-5b7f-4b7e-9695-c8125cdcffae"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/59ffb213-5b7f-4b7e-9695-c8125cdcffae" region = 59ffb213-5b7f-4b7e-9695-c8125cdcffae
412026-05-27T21:13:20.963ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "59ffb213-5b7f-4b7e-9695-c8125cdcffae" "--data" "/data/regions/59ffb213-5b7f-4b7e-9695-c8125cdcffae" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 59ffb213-5b7f-4b7e-9695-c8125cdcffae
422026-05-27T21:13:21.014ZINFOcrucible-agent (worker): region files created ok region = 59ffb213-5b7f-4b7e-9695-c8125cdcffae
432026-05-27T21:13:21.014ZINFOcrucible-agent (datafile): region 59ffb213-5b7f-4b7e-9695-c8125cdcffae state: Requested -> Created
442026-05-27T21:13:21.014ZINFOcrucible-agent (worker): applying SMF actions post create...
452026-05-27T21:13:21.014ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-59ffb213-5b7f-4b7e-9695-c8125cdcffae
462026-05-27T21:13:21.016ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-59ffb213-5b7f-4b7e-9695-c8125cdcffae
472026-05-27T21:13:21.019ZINFOcrucible-agent (worker): creating config property group
482026-05-27T21:13:21.021ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-59ffb213-5b7f-4b7e-9695-c8125cdcffae
492026-05-27T21:13:21.021ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/59ffb213-5b7f-4b7e-9695-c8125cdcffae
502026-05-27T21:13:21.021ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
512026-05-27T21:13:21.021ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
522026-05-27T21:13:21.021ZINFOcrucible-agent (worker): commit
532026-05-27T21:13:21.023ZINFOcrucible-agent (worker): ok!
542026-05-27T21:13:21.026ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/86e12674-1371-4728-9221-3a0fb9722ae1
552026-05-27T21:13:21.026ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
562026-05-27T21:13:21.026ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
572026-05-27T21:13:21.026ZINFOcrucible-agent (worker): reconfig required
582026-05-27T21:13:21.026ZINFOcrucible-agent (worker): using existing config property group
592026-05-27T21:13:21.026ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-86e12674-1371-4728-9221-3a0fb9722ae1
602026-05-27T21:13:21.026ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/86e12674-1371-4728-9221-3a0fb9722ae1
612026-05-27T21:13:21.026ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
622026-05-27T21:13:21.026ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
632026-05-27T21:13:21.026ZINFOcrucible-agent (worker): commit
642026-05-27T21:13:21.030ZINFOcrucible-agent (worker): ok!
652026-05-27T21:13:21.031ZINFOcrucible-agent (worker): SMF ok!
662026-05-27T21:13:21.051ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51453 req_id = 153b2450-00c1-4b49-8e7d-a73c14085b8c response_code = 200 uri = /crucible/0/regions
672026-05-27T21:13:45.836ZINFOcrucible-agent (dropshot): request completed latency_us = 72 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51453 req_id = 69d4cd90-90af-4091-93ec-f1ba9960f504 response_code = 200 uri = /crucible/0/regions/59ffb213-5b7f-4b7e-9695-c8125cdcffae
682026-05-27T21:13:45.840ZINFOcrucible-agent (datafile): path is "/data/regions/59ffb213-5b7f-4b7e-9695-c8125cdcffae"
692026-05-27T21:13:45.848ZINFOcrucible-agent (datafile): dataset is oxp_a944f5b5-20fa-47fb-9f8a-59ed228bb8db/crucible/regions/59ffb213-5b7f-4b7e-9695-c8125cdcffae
702026-05-27T21:13:45.856ZINFOcrucible-agent (datafile): region 59ffb213-5b7f-4b7e-9695-c8125cdcffae state: Created -> Tombstoned
712026-05-27T21:13:45.856ZINFOcrucible-agent (dropshot): request completed latency_us = 16177 local_addr = [fd00:1122:3344:101::14]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:51453 req_id = 0e5f01b2-f2f1-4fae-bfc8-db88430e8c46 response_code = 204 uri = /crucible/0/regions/59ffb213-5b7f-4b7e-9695-c8125cdcffae
722026-05-27T21:13:45.856ZINFOcrucible-agent (worker): applying SMF actions before removal...
732026-05-27T21:13:45.856ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-59ffb213-5b7f-4b7e-9695-c8125cdcffae (instance states: (Some(Online), None))
742026-05-27T21:13:45.859ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/86e12674-1371-4728-9221-3a0fb9722ae1
752026-05-27T21:13:45.859ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
762026-05-27T21:13:45.859ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
772026-05-27T21:13:45.859ZINFOcrucible-agent (worker): reconfig required
782026-05-27T21:13:45.859ZINFOcrucible-agent (worker): using existing config property group
792026-05-27T21:13:45.859ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-86e12674-1371-4728-9221-3a0fb9722ae1
802026-05-27T21:13:45.860ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/86e12674-1371-4728-9221-3a0fb9722ae1
812026-05-27T21:13:45.860ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51453 req_id = 940b9ec1-eff9-4e9b-a439-0d903003f289 response_code = 200 uri = /crucible/0/regions/59ffb213-5b7f-4b7e-9695-c8125cdcffae
822026-05-27T21:13:45.860ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
832026-05-27T21:13:45.860ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
842026-05-27T21:13:45.860ZINFOcrucible-agent (worker): commit
852026-05-27T21:13:45.862ZINFOcrucible-agent (worker): ok!
862026-05-27T21:13:45.863ZINFOcrucible-agent (worker): SMF ok!
872026-05-27T21:13:45.871ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_a944f5b5-20fa-47fb-9f8a-59ed228bb8db/crucible/regions/59ffb213-5b7f-4b7e-9695-c8125cdcffae" region = 59ffb213-5b7f-4b7e-9695-c8125cdcffae
882026-05-27T21:13:45.925ZINFOcrucible-agent (datafile): region 59ffb213-5b7f-4b7e-9695-c8125cdcffae state: Tombstoned -> Destroyed
892026-05-27T21:13:45.937ZINFOcrucible-agent (dropshot): request completed latency_us = 49 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51453 req_id = a742b8fd-899e-4588-bcd9-8e4d5c3f045c response_code = 200 uri = /crucible/0/regions/59ffb213-5b7f-4b7e-9695-c8125cdcffae