Max level shown:
1 [ Dec 31 23:16:04 Enabled. ]
2 [ Dec 31 23:16:04 Rereading configuration. ]
3 [ Dec 31 23:16:07 Rereading configuration. ]
4 [ Dec 31 23:16:11 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible -l [fd00:1122:3344:101::18]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-12-31T23:16:12.267ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 7103cd3a3d7b0112d2949dd135db06fef0c156bb Commit timestamp: 2025-11-10T22:24:53.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72025-12-31T23:16:12.268ZINFOcrucible-agent: dataset: "oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible"
82025-12-31T23:16:12.268ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::18]:32345
92025-12-31T23:16:12.268ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102025-12-31T23:16:12.284ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112025-12-31T23:16:12.363ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::18]:32345
122025-12-31T23:16:42.021ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:38540
132025-12-31T23:16:42.053ZINFOcrucible-agent (datafile): region df0a7517-008e-406b-8ba9-0400efff055a state: Requested
142025-12-31T23:16:42.055ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152025-12-31T23:16:42.056ZINFOcrucible-agent (dropshot): request completed latency_us = 16804 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38540 req_id = 18c42f64-059f-4504-bdf1-31855a76c887 response_code = 200 uri = /crucible/0/regions
162025-12-31T23:16:42.064ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible/regions/df0a7517-008e-406b-8ba9-0400efff055a
172025-12-31T23:16:42.064ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible/regions/df0a7517-008e-406b-8ba9-0400efff055a
182025-12-31T23:16:42.132ZINFOcrucible-agent (worker): creating region Region { id: RegionId("df0a7517-008e-406b-8ba9-0400efff055a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/df0a7517-008e-406b-8ba9-0400efff055a" region = df0a7517-008e-406b-8ba9-0400efff055a
192025-12-31T23:16:42.134ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "df0a7517-008e-406b-8ba9-0400efff055a" "--data" "/data/regions/df0a7517-008e-406b-8ba9-0400efff055a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = df0a7517-008e-406b-8ba9-0400efff055a
202025-12-31T23:16:42.396ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38540 req_id = 1d4ba7d9-3027-4319-8c8c-baaa326cc74a response_code = 200 uri = /crucible/0/regions
212025-12-31T23:16:42.509ZINFOcrucible-agent (worker): region files created ok region = df0a7517-008e-406b-8ba9-0400efff055a
222025-12-31T23:16:42.509ZINFOcrucible-agent (datafile): region df0a7517-008e-406b-8ba9-0400efff055a state: Requested -> Created
232025-12-31T23:16:42.510ZINFOcrucible-agent (worker): applying SMF actions post create...
242025-12-31T23:16:42.510ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-df0a7517-008e-406b-8ba9-0400efff055a
252025-12-31T23:16:42.518ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-df0a7517-008e-406b-8ba9-0400efff055a
262025-12-31T23:16:42.532ZINFOcrucible-agent (worker): creating config property group
272025-12-31T23:16:42.534ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-df0a7517-008e-406b-8ba9-0400efff055a
282025-12-31T23:16:42.535ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/df0a7517-008e-406b-8ba9-0400efff055a
292025-12-31T23:16:42.535ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302025-12-31T23:16:42.535ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
312025-12-31T23:16:42.535ZINFOcrucible-agent (worker): commit
322025-12-31T23:16:42.540ZINFOcrucible-agent (worker): ok!
332025-12-31T23:16:42.545ZINFOcrucible-agent (worker): SMF ok!
342025-12-31T23:16:42.982ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38540 req_id = ab206582-ff55-4b5a-8788-2c5a8dbc3483 response_code = 200 uri = /crucible/0/regions
352025-12-31T23:28:10.045ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:54978
362025-12-31T23:28:10.045ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54978 req_id = 38fa4d1a-09e2-4e98-9678-284edfa21d5c response_code = 200 uri = /crucible/0/regions/df0a7517-008e-406b-8ba9-0400efff055a
372025-12-31T23:28:10.053ZINFOcrucible-agent (datafile): path is "/data/regions/df0a7517-008e-406b-8ba9-0400efff055a"
382025-12-31T23:28:10.060ZINFOcrucible-agent (datafile): dataset is oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible/regions/df0a7517-008e-406b-8ba9-0400efff055a
392025-12-31T23:28:10.066ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible/regions/df0a7517-008e-406b-8ba9-0400efff055a@8e768e29-734d-4cdc-8581-731a9ddac360
402025-12-31T23:28:10.066ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible/regions/df0a7517-008e-406b-8ba9-0400efff055a", "8e768e29-734d-4cdc-8581-731a9ddac360"]
412025-12-31T23:28:10.082ZINFOcrucible-agent (dropshot): request completed latency_us = 29088 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54978 req_id = 067e4078-da47-464b-8132-d65e7fdd4df7 response_code = 200 uri = /crucible/0/regions/df0a7517-008e-406b-8ba9-0400efff055a/snapshots/8e768e29-734d-4cdc-8581-731a9ddac360
422025-12-31T23:28:10.098ZINFOcrucible-agent (datafile): path is "/data/regions/df0a7517-008e-406b-8ba9-0400efff055a"
432025-12-31T23:28:10.103ZINFOcrucible-agent (datafile): dataset is oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible/regions/df0a7517-008e-406b-8ba9-0400efff055a
442025-12-31T23:28:10.110ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible/regions/df0a7517-008e-406b-8ba9-0400efff055a@8e768e29-734d-4cdc-8581-731a9ddac360
452025-12-31T23:28:10.110ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible/regions/df0a7517-008e-406b-8ba9-0400efff055a", "8e768e29-734d-4cdc-8581-731a9ddac360"]
462025-12-31T23:28:10.122ZINFOcrucible-agent (datafile): requesting running snapshot df0a7517-008e-406b-8ba9-0400efff055a-8e768e29-734d-4cdc-8581-731a9ddac360 state: Requested
472025-12-31T23:28:10.123ZINFOcrucible-agent (dropshot): request completed latency_us = 25417 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54978 req_id = c1b841fe-545c-4bf5-aa7a-e89ebd90bf1c response_code = 200 uri = /crucible/0/regions/df0a7517-008e-406b-8ba9-0400efff055a/snapshots/8e768e29-734d-4cdc-8581-731a9ddac360/run
482025-12-31T23:28:10.123ZINFOcrucible-agent (worker): applying SMF actions for region df0a7517-008e-406b-8ba9-0400efff055a running snapshot 8e768e29-734d-4cdc-8581-731a9ddac360 (state Requested)...
492025-12-31T23:28:10.145ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-df0a7517-008e-406b-8ba9-0400efff055a-8e768e29-734d-4cdc-8581-731a9ddac360
502025-12-31T23:28:10.157ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-df0a7517-008e-406b-8ba9-0400efff055a-8e768e29-734d-4cdc-8581-731a9ddac360
512025-12-31T23:28:10.192ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-df0a7517-008e-406b-8ba9-0400efff055a-8e768e29-734d-4cdc-8581-731a9ddac360
522025-12-31T23:28:10.193ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/df0a7517-008e-406b-8ba9-0400efff055a/.zfs/snapshot/8e768e29-734d-4cdc-8581-731a9ddac360
532025-12-31T23:28:10.193ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
542025-12-31T23:28:10.193ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
552025-12-31T23:28:10.193ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
562025-12-31T23:28:10.193ZINFOcrucible-agent (worker): commit
572025-12-31T23:28:10.200ZINFOcrucible-agent (worker): ok!
582025-12-31T23:28:10.209ZINFOcrucible-agent (worker): SMF ok!
592025-12-31T23:28:10.209ZINFOcrucible-agent (datafile): region df0a7517-008e-406b-8ba9-0400efff055a running snapshot 8e768e29-734d-4cdc-8581-731a9ddac360 state: Requested -> Created
602025-12-31T23:28:10.366ZINFOcrucible-agent (datafile): path is "/data/regions/df0a7517-008e-406b-8ba9-0400efff055a"
612025-12-31T23:28:10.373ZINFOcrucible-agent (datafile): dataset is oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible/regions/df0a7517-008e-406b-8ba9-0400efff055a
622025-12-31T23:28:10.380ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible/regions/df0a7517-008e-406b-8ba9-0400efff055a@8e768e29-734d-4cdc-8581-731a9ddac360
632025-12-31T23:28:10.380ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible/regions/df0a7517-008e-406b-8ba9-0400efff055a", "8e768e29-734d-4cdc-8581-731a9ddac360"]
642025-12-31T23:28:10.385ZINFOcrucible-agent (dropshot): request completed latency_us = 19120 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54978 req_id = 6b8b8160-6eee-40ee-b17a-f23cd984a5de response_code = 200 uri = /crucible/0/regions/df0a7517-008e-406b-8ba9-0400efff055a/snapshots/8e768e29-734d-4cdc-8581-731a9ddac360/run
652025-12-31T23:28:16.454ZINFOcrucible-agent (datafile): region a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6 state: Requested
662025-12-31T23:28:16.454ZINFOcrucible-agent (dropshot): request completed latency_us = 231 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54978 req_id = 5acb04e2-0ad3-4876-bc18-0de9ca55735f response_code = 200 uri = /crucible/0/regions
672025-12-31T23:28:16.454ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
682025-12-31T23:28:16.462ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible/regions/a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6
692025-12-31T23:28:16.462ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible/regions/a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6
702025-12-31T23:28:16.573ZINFOcrucible-agent (worker): creating region Region { id: RegionId("a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6" region = a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6
712025-12-31T23:28:16.573ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6" "--data" "/data/regions/a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6
722025-12-31T23:28:16.690ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54978 req_id = a434cccc-52e0-4413-bdc5-05c125df929c response_code = 200 uri = /crucible/0/regions
732025-12-31T23:28:16.738ZINFOcrucible-agent (worker): region files created ok region = a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6
742025-12-31T23:28:16.738ZINFOcrucible-agent (datafile): region a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6 state: Requested -> Created
752025-12-31T23:28:16.738ZINFOcrucible-agent (worker): applying SMF actions post create...
762025-12-31T23:28:16.739ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6
772025-12-31T23:28:16.747ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6
782025-12-31T23:28:16.772ZINFOcrucible-agent (worker): creating config property group
792025-12-31T23:28:16.784ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6
802025-12-31T23:28:16.784ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6
812025-12-31T23:28:16.784ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
822025-12-31T23:28:16.784ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
832025-12-31T23:28:16.784ZINFOcrucible-agent (worker): commit
842025-12-31T23:28:16.799ZINFOcrucible-agent (worker): ok!
852025-12-31T23:28:16.829ZINFOcrucible-agent (worker): SMF ok!
862025-12-31T23:28:17.105ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54978 req_id = c0371d0c-cb69-41ac-851c-56aa1313ecbf response_code = 200 uri = /crucible/0/regions
872025-12-31T23:28:46.444ZINFOcrucible-agent (dropshot): request completed latency_us = 63 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54978 req_id = 671f8fec-94cb-4317-8640-34616337f079 response_code = 200 uri = /crucible/0/regions/a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6
882025-12-31T23:28:46.447ZINFOcrucible-agent (datafile): path is "/data/regions/a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6"
892025-12-31T23:28:46.457ZINFOcrucible-agent (datafile): dataset is oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible/regions/a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6
902025-12-31T23:28:46.465ZINFOcrucible-agent (datafile): region a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6 state: Created -> Tombstoned
912025-12-31T23:28:46.466ZINFOcrucible-agent (dropshot): request completed latency_us = 18351 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:54978 req_id = 832fbf3a-41c4-4b96-a9fa-12dcd7c8bb21 response_code = 204 uri = /crucible/0/regions/a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6
922025-12-31T23:28:46.466ZINFOcrucible-agent (worker): applying SMF actions before removal...
932025-12-31T23:28:46.466ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6 (instance states: (Some(Online), None))
942025-12-31T23:28:46.469ZINFOcrucible-agent (dropshot): request completed latency_us = 33 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54978 req_id = b36bdd5e-858f-4f98-a926-e1130e48a4ba response_code = 200 uri = /crucible/0/regions/a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6
952025-12-31T23:28:46.482ZINFOcrucible-agent (worker): SMF ok!
962025-12-31T23:28:46.490ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_9b81f321-b39d-43d5-90a4-24209186d89f/crucible/regions/a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6" region = a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6
972025-12-31T23:28:46.528ZINFOcrucible-agent (dropshot): request completed latency_us = 39 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54978 req_id = d4be9de7-d2b2-4d53-8ef5-d438f1287425 response_code = 200 uri = /crucible/0/regions/a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6
982025-12-31T23:28:46.550ZINFOcrucible-agent (datafile): region a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6 state: Tombstoned -> Destroyed
992025-12-31T23:28:46.649ZINFOcrucible-agent (dropshot): request completed latency_us = 41 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54978 req_id = c9975d59-0480-4c5a-a857-2b7743ef6ec3 response_code = 200 uri = /crucible/0/regions/a79c6b6c-d0c6-40d6-b1f6-9ce1242da7d6