Max level shown:
1 [ Aug 26 21:45:41 Enabled. ]
2 [ Aug 26 21:45:41 Rereading configuration. ]
3 [ Aug 26 21:45:45 Rereading configuration. ]
4 [ Aug 26 21:45:53 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_ee4aef7e-79f7-4692-bb40-d6c368fe7a80/crucible -l [fd00:1122:3344:101::19]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-08-26T21:45:54.122ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 779775d5130ff7a4836f52f48b7e64d1479ee104 Commit timestamp: 2025-08-09T00:27:59.000000000Z branch: main rustc: 1.86.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72025-08-26T21:45:54.125ZINFOcrucible-agent: dataset: "oxp_ee4aef7e-79f7-4692-bb40-d6c368fe7a80/crucible"
82025-08-26T21:45:54.125ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::19]:32345
92025-08-26T21:45:54.125ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102025-08-26T21:45:54.142ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112025-08-26T21:45:54.217ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::19]:32345
122025-08-26T21:56:42.901ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::b]:47844
132025-08-26T21:56:42.947ZINFOcrucible-agent (datafile): region 45ea8804-b169-4690-a678-76035a22638e state: Requested
142025-08-26T21:56:42.955ZINFOcrucible-agent (dropshot): request completed latency_us = 25738 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:47844 req_id = 664775d0-3804-40fd-90c1-2ba45cad9d8c response_code = 200 uri = /crucible/0/regions
152025-08-26T21:56:42.960ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
162025-08-26T21:56:42.973ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_ee4aef7e-79f7-4692-bb40-d6c368fe7a80/crucible/regions/45ea8804-b169-4690-a678-76035a22638e
172025-08-26T21:56:42.973ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_ee4aef7e-79f7-4692-bb40-d6c368fe7a80/crucible/regions/45ea8804-b169-4690-a678-76035a22638e
182025-08-26T21:56:43.022ZINFOcrucible-agent (worker): creating region Region { id: RegionId("45ea8804-b169-4690-a678-76035a22638e"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/45ea8804-b169-4690-a678-76035a22638e" region = 45ea8804-b169-4690-a678-76035a22638e
192025-08-26T21:56:43.024ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "45ea8804-b169-4690-a678-76035a22638e" "--data" "/data/regions/45ea8804-b169-4690-a678-76035a22638e" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 45ea8804-b169-4690-a678-76035a22638e
202025-08-26T21:56:43.146ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:47844 req_id = 1173e17b-189d-4521-9df3-8de9e792dd22 response_code = 200 uri = /crucible/0/regions
212025-08-26T21:56:43.338ZINFOcrucible-agent (worker): region files created ok region = 45ea8804-b169-4690-a678-76035a22638e
222025-08-26T21:56:43.338ZINFOcrucible-agent (datafile): region 45ea8804-b169-4690-a678-76035a22638e state: Requested -> Created
232025-08-26T21:56:43.338ZINFOcrucible-agent (worker): applying SMF actions post create...
242025-08-26T21:56:43.339ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-45ea8804-b169-4690-a678-76035a22638e
252025-08-26T21:56:43.350ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-45ea8804-b169-4690-a678-76035a22638e
262025-08-26T21:56:43.371ZINFOcrucible-agent (worker): creating config property group
272025-08-26T21:56:43.374ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-45ea8804-b169-4690-a678-76035a22638e
282025-08-26T21:56:43.375ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/45ea8804-b169-4690-a678-76035a22638e
292025-08-26T21:56:43.384ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302025-08-26T21:56:43.384ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
312025-08-26T21:56:43.384ZINFOcrucible-agent (worker): commit
322025-08-26T21:56:43.394ZINFOcrucible-agent (worker): ok!
332025-08-26T21:56:43.398ZINFOcrucible-agent (worker): SMF ok!
342025-08-26T21:56:43.543ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:47844 req_id = 719590fd-3595-41c5-a93c-d7dc276ab175 response_code = 200 uri = /crucible/0/regions
352025-08-26T21:56:49.797ZINFOcrucible-agent (datafile): region ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001 state: Requested
362025-08-26T21:56:49.797ZINFOcrucible-agent (dropshot): request completed latency_us = 256 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:47844 req_id = 25c47733-d532-40f4-903d-53002859812e response_code = 200 uri = /crucible/0/regions
372025-08-26T21:56:49.797ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
382025-08-26T21:56:49.808ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_ee4aef7e-79f7-4692-bb40-d6c368fe7a80/crucible/regions/ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001
392025-08-26T21:56:49.808ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_ee4aef7e-79f7-4692-bb40-d6c368fe7a80/crucible/regions/ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001
402025-08-26T21:56:49.857ZINFOcrucible-agent (worker): creating region Region { id: RegionId("ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001" region = ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001
412025-08-26T21:56:49.857ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001" "--data" "/data/regions/ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001
422025-08-26T21:56:49.934ZINFOcrucible-agent (worker): region files created ok region = ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001
432025-08-26T21:56:49.934ZINFOcrucible-agent (datafile): region ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001 state: Requested -> Created
442025-08-26T21:56:49.934ZINFOcrucible-agent (worker): applying SMF actions post create...
452025-08-26T21:56:49.935ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/45ea8804-b169-4690-a678-76035a22638e
462025-08-26T21:56:49.935ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
472025-08-26T21:56:49.936ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::19
482025-08-26T21:56:49.936ZINFOcrucible-agent (worker): reconfig required
492025-08-26T21:56:49.936ZINFOcrucible-agent (worker): using existing config property group
502025-08-26T21:56:49.936ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-45ea8804-b169-4690-a678-76035a22638e
512025-08-26T21:56:49.936ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/45ea8804-b169-4690-a678-76035a22638e
522025-08-26T21:56:49.936ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
532025-08-26T21:56:49.936ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
542025-08-26T21:56:49.936ZINFOcrucible-agent (worker): commit
552025-08-26T21:56:49.939ZINFOcrucible-agent (worker): ok!
562025-08-26T21:56:49.940ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001
572025-08-26T21:56:49.942ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001
582025-08-26T21:56:49.950ZINFOcrucible-agent (worker): creating config property group
592025-08-26T21:56:49.953ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001
602025-08-26T21:56:49.953ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001
612025-08-26T21:56:49.953ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
622025-08-26T21:56:49.953ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
632025-08-26T21:56:49.953ZINFOcrucible-agent (worker): commit
642025-08-26T21:56:49.957ZINFOcrucible-agent (worker): ok!
652025-08-26T21:56:49.963ZINFOcrucible-agent (worker): SMF ok!
662025-08-26T21:56:50.164ZINFOcrucible-agent (dropshot): request completed latency_us = 101 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:47844 req_id = 78773d82-825b-4d13-ba6b-7cb0bb9fb6bc response_code = 200 uri = /crucible/0/regions
672025-08-26T21:57:14.708ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:47844 req_id = 3956b6be-0bb6-478c-8179-c6041c96e176 response_code = 200 uri = /crucible/0/regions/ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001
682025-08-26T21:57:14.710ZINFOcrucible-agent (datafile): path is "/data/regions/ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001"
692025-08-26T21:57:14.724ZINFOcrucible-agent (datafile): dataset is oxp_ee4aef7e-79f7-4692-bb40-d6c368fe7a80/crucible/regions/ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001
702025-08-26T21:57:14.737ZINFOcrucible-agent (datafile): region ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001 state: Created -> Tombstoned
712025-08-26T21:57:14.737ZINFOcrucible-agent (worker): applying SMF actions before removal...
722025-08-26T21:57:14.737ZINFOcrucible-agent (dropshot): request completed latency_us = 27129 local_addr = [fd00:1122:3344:101::19]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:47844 req_id = b3ae7d2b-77b4-4eb8-a236-06f9d8ca780b response_code = 204 uri = /crucible/0/regions/ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001
732025-08-26T21:57:14.738ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001 (instance states: (Some(Online), None))
742025-08-26T21:57:14.741ZINFOcrucible-agent (dropshot): request completed latency_us = 68 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:47844 req_id = 3e6ef8ac-2606-4c99-8caf-868a10bd3527 response_code = 200 uri = /crucible/0/regions/ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001
752025-08-26T21:57:14.744ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/45ea8804-b169-4690-a678-76035a22638e
762025-08-26T21:57:14.744ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
772025-08-26T21:57:14.744ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::19
782025-08-26T21:57:14.744ZINFOcrucible-agent (worker): reconfig required
792025-08-26T21:57:14.744ZINFOcrucible-agent (worker): using existing config property group
802025-08-26T21:57:14.744ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-45ea8804-b169-4690-a678-76035a22638e
812025-08-26T21:57:14.745ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/45ea8804-b169-4690-a678-76035a22638e
822025-08-26T21:57:14.745ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
832025-08-26T21:57:14.745ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
842025-08-26T21:57:14.745ZINFOcrucible-agent (worker): commit
852025-08-26T21:57:14.749ZINFOcrucible-agent (worker): ok!
862025-08-26T21:57:14.749ZINFOcrucible-agent (worker): SMF ok!
872025-08-26T21:57:14.761ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_ee4aef7e-79f7-4692-bb40-d6c368fe7a80/crucible/regions/ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001" region = ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001
882025-08-26T21:57:14.800ZINFOcrucible-agent (dropshot): request completed latency_us = 92 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:47844 req_id = 9423484f-c3f9-4a64-9144-321149f47433 response_code = 200 uri = /crucible/0/regions/ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001
892025-08-26T21:57:14.829ZINFOcrucible-agent (datafile): region ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001 state: Tombstoned -> Destroyed
902025-08-26T21:57:14.929ZINFOcrucible-agent (dropshot): request completed latency_us = 34 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:47844 req_id = 396fca25-864e-45f3-9057-40dc0fad0ef3 response_code = 200 uri = /crucible/0/regions/ab617fbd-3dd3-4f8e-9a5a-dc5d7aa90001