Max level shown:
1 [ Jun 25 01:01:04 Enabled. ]
2 [ Jun 25 01:01:04 Rereading configuration. ]
3 [ Jun 25 01:01:06 Rereading configuration. ]
4 [ Jun 25 01:01:09 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_950e579c-96c4-4dc8-bfb1-eef131a5818b/crucible -l [fd00:1122:3344:101::18]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-06-25T01:01:09.655ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: bd9a0e2abe6b6b89aec8c85f4ee57474144ed150 Commit timestamp: 2026-05-14T15:48:00.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-06-25T01:01:09.655ZINFOcrucible-agent: dataset: "oxp_950e579c-96c4-4dc8-bfb1-eef131a5818b/crucible"
82026-06-25T01:01:09.655ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::18]:32345
92026-06-25T01:01:09.655ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-06-25T01:01:09.675ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-06-25T01:01:09.760ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::18]:32345
122026-06-25T01:02:12.801ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::b]:58959
132026-06-25T01:02:12.824ZINFOcrucible-agent (datafile): region d94b7b0c-674f-4ae6-b29f-73d59e4d9f73 state: Requested
142026-06-25T01:02:12.830ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152026-06-25T01:02:12.830ZINFOcrucible-agent (dropshot): request completed latency_us = 13155 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:58959 req_id = 3c95a472-c88c-45d7-97fc-d1f5edce11d0 response_code = 200 uri = /crucible/0/regions
162026-06-25T01:02:12.837ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_950e579c-96c4-4dc8-bfb1-eef131a5818b/crucible/regions/d94b7b0c-674f-4ae6-b29f-73d59e4d9f73
172026-06-25T01:02:12.837ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_950e579c-96c4-4dc8-bfb1-eef131a5818b/crucible/regions/d94b7b0c-674f-4ae6-b29f-73d59e4d9f73
182026-06-25T01:02:12.884ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d94b7b0c-674f-4ae6-b29f-73d59e4d9f73"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d94b7b0c-674f-4ae6-b29f-73d59e4d9f73" region = d94b7b0c-674f-4ae6-b29f-73d59e4d9f73
192026-06-25T01:02:12.884ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d94b7b0c-674f-4ae6-b29f-73d59e4d9f73" "--data" "/data/regions/d94b7b0c-674f-4ae6-b29f-73d59e4d9f73" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = d94b7b0c-674f-4ae6-b29f-73d59e4d9f73
202026-06-25T01:02:13.103ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:58959 req_id = a704fb76-fb1d-421c-90f7-a414ef7b574c response_code = 200 uri = /crucible/0/regions
212026-06-25T01:02:13.200ZINFOcrucible-agent (worker): region files created ok region = d94b7b0c-674f-4ae6-b29f-73d59e4d9f73
222026-06-25T01:02:13.200ZINFOcrucible-agent (datafile): region d94b7b0c-674f-4ae6-b29f-73d59e4d9f73 state: Requested -> Created
232026-06-25T01:02:13.201ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-06-25T01:02:13.203ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d94b7b0c-674f-4ae6-b29f-73d59e4d9f73
252026-06-25T01:02:13.208ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d94b7b0c-674f-4ae6-b29f-73d59e4d9f73
262026-06-25T01:02:13.222ZINFOcrucible-agent (worker): creating config property group
272026-06-25T01:02:13.223ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d94b7b0c-674f-4ae6-b29f-73d59e4d9f73
282026-06-25T01:02:13.224ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d94b7b0c-674f-4ae6-b29f-73d59e4d9f73
292026-06-25T01:02:13.224ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-06-25T01:02:13.224ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
312026-06-25T01:02:13.224ZINFOcrucible-agent (worker): commit
322026-06-25T01:02:13.235ZINFOcrucible-agent (worker): ok!
332026-06-25T01:02:13.237ZINFOcrucible-agent (worker): SMF ok!
342026-06-25T01:02:13.538ZINFOcrucible-agent (dropshot): request completed latency_us = 97 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:58959 req_id = 7ffeca3d-9138-4e84-b494-2bf9dabd7786 response_code = 200 uri = /crucible/0/regions
352026-06-25T01:02:14.051ZINFOcrucible-agent (dropshot): request completed latency_us = 152 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:58959 req_id = c3a7713e-c9fe-4497-9eb2-0f18e5ddf2f0 response_code = 200 uri = /crucible/0/regions/d94b7b0c-674f-4ae6-b29f-73d59e4d9f73
362026-06-25T01:02:14.054ZINFOcrucible-agent (datafile): path is "/data/regions/d94b7b0c-674f-4ae6-b29f-73d59e4d9f73"
372026-06-25T01:02:14.062ZINFOcrucible-agent (datafile): dataset is oxp_950e579c-96c4-4dc8-bfb1-eef131a5818b/crucible/regions/d94b7b0c-674f-4ae6-b29f-73d59e4d9f73
382026-06-25T01:02:14.068ZINFOcrucible-agent (datafile): region d94b7b0c-674f-4ae6-b29f-73d59e4d9f73 state: Created -> Tombstoned
392026-06-25T01:02:14.068ZINFOcrucible-agent (worker): applying SMF actions before removal...
402026-06-25T01:02:14.068ZINFOcrucible-agent (dropshot): request completed latency_us = 14000 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:58959 req_id = 0fe79b6d-8e31-4214-a418-5ff90d93f8d3 response_code = 204 uri = /crucible/0/regions/d94b7b0c-674f-4ae6-b29f-73d59e4d9f73
412026-06-25T01:02:14.069ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-d94b7b0c-674f-4ae6-b29f-73d59e4d9f73 (instance states: (Some(Online), None))
422026-06-25T01:02:14.072ZINFOcrucible-agent (worker): SMF ok!
432026-06-25T01:02:14.078ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:58959 req_id = a233c783-ab9b-47d5-93a9-92741920496a response_code = 200 uri = /crucible/0/regions/d94b7b0c-674f-4ae6-b29f-73d59e4d9f73
442026-06-25T01:02:14.081ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_950e579c-96c4-4dc8-bfb1-eef131a5818b/crucible/regions/d94b7b0c-674f-4ae6-b29f-73d59e4d9f73" region = d94b7b0c-674f-4ae6-b29f-73d59e4d9f73
452026-06-25T01:02:14.134ZINFOcrucible-agent (datafile): region d94b7b0c-674f-4ae6-b29f-73d59e4d9f73 state: Tombstoned -> Destroyed
462026-06-25T01:02:14.215ZINFOcrucible-agent (dropshot): request completed latency_us = 63 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:58959 req_id = 4013752e-6e69-4225-8abd-39d754b6d02a response_code = 200 uri = /crucible/0/regions/d94b7b0c-674f-4ae6-b29f-73d59e4d9f73
472026-06-25T01:13:15.627ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::b]:59559
482026-06-25T01:13:15.627ZINFOcrucible-agent (datafile): region 2e646ef0-6dfc-43af-8179-1f5e01ccc301 state: Requested
492026-06-25T01:13:15.631ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
502026-06-25T01:13:15.631ZINFOcrucible-agent (dropshot): request completed latency_us = 3967 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:59559 req_id = 3b24e649-c7e0-4084-86fd-9068d7810865 response_code = 200 uri = /crucible/0/regions
512026-06-25T01:13:15.639ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_950e579c-96c4-4dc8-bfb1-eef131a5818b/crucible/regions/2e646ef0-6dfc-43af-8179-1f5e01ccc301
522026-06-25T01:13:15.639ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_950e579c-96c4-4dc8-bfb1-eef131a5818b/crucible/regions/2e646ef0-6dfc-43af-8179-1f5e01ccc301
532026-06-25T01:13:15.686ZINFOcrucible-agent (worker): creating region Region { id: RegionId("2e646ef0-6dfc-43af-8179-1f5e01ccc301"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/2e646ef0-6dfc-43af-8179-1f5e01ccc301" region = 2e646ef0-6dfc-43af-8179-1f5e01ccc301
542026-06-25T01:13:15.686ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "2e646ef0-6dfc-43af-8179-1f5e01ccc301" "--data" "/data/regions/2e646ef0-6dfc-43af-8179-1f5e01ccc301" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 2e646ef0-6dfc-43af-8179-1f5e01ccc301
552026-06-25T01:13:15.730ZINFOcrucible-agent (worker): region files created ok region = 2e646ef0-6dfc-43af-8179-1f5e01ccc301
562026-06-25T01:13:15.730ZINFOcrucible-agent (datafile): region 2e646ef0-6dfc-43af-8179-1f5e01ccc301 state: Requested -> Created
572026-06-25T01:13:15.730ZINFOcrucible-agent (worker): applying SMF actions post create...
582026-06-25T01:13:15.732ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-2e646ef0-6dfc-43af-8179-1f5e01ccc301
592026-06-25T01:13:15.735ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-2e646ef0-6dfc-43af-8179-1f5e01ccc301
602026-06-25T01:13:15.745ZINFOcrucible-agent (worker): creating config property group
612026-06-25T01:13:15.747ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-2e646ef0-6dfc-43af-8179-1f5e01ccc301
622026-06-25T01:13:15.747ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/2e646ef0-6dfc-43af-8179-1f5e01ccc301
632026-06-25T01:13:15.747ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
642026-06-25T01:13:15.747ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
652026-06-25T01:13:15.747ZINFOcrucible-agent (worker): commit
662026-06-25T01:13:15.749ZINFOcrucible-agent (worker): ok!
672026-06-25T01:13:15.752ZINFOcrucible-agent (worker): SMF ok!
682026-06-25T01:13:16.001ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:59559 req_id = d015e17b-b0bd-4739-9e13-44a138ab0cf5 response_code = 200 uri = /crucible/0/regions
692026-06-25T01:13:22.070ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:59017
702026-06-25T01:13:22.074ZINFOcrucible-agent (dropshot): request completed error_message_external = Not Found error_message_internal = region RegionId("006f31eb-3867-4dd4-8c81-c85062376609") not found latency_us = 3250 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:59017 req_id = 725075f4-b992-4d21-8202-600617001830 response_code = 404 uri = /crucible/0/regions/006f31eb-3867-4dd4-8c81-c85062376609
712026-06-25T01:13:22.079ZINFOcrucible-agent (datafile): region 006f31eb-3867-4dd4-8c81-c85062376609 state: Requested
722026-06-25T01:13:22.079ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
732026-06-25T01:13:22.079ZINFOcrucible-agent (dropshot): request completed latency_us = 350 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:59559 req_id = 436a0cc0-3af9-489d-98b5-1c8e512d9b08 response_code = 200 uri = /crucible/0/regions
742026-06-25T01:13:22.089ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_950e579c-96c4-4dc8-bfb1-eef131a5818b/crucible/regions/006f31eb-3867-4dd4-8c81-c85062376609
752026-06-25T01:13:22.089ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_950e579c-96c4-4dc8-bfb1-eef131a5818b/crucible/regions/006f31eb-3867-4dd4-8c81-c85062376609
762026-06-25T01:13:22.170ZINFOcrucible-agent (worker): creating region Region { id: RegionId("006f31eb-3867-4dd4-8c81-c85062376609"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/006f31eb-3867-4dd4-8c81-c85062376609" region = 006f31eb-3867-4dd4-8c81-c85062376609
772026-06-25T01:13:22.170ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "006f31eb-3867-4dd4-8c81-c85062376609" "--data" "/data/regions/006f31eb-3867-4dd4-8c81-c85062376609" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 006f31eb-3867-4dd4-8c81-c85062376609
782026-06-25T01:13:22.262ZINFOcrucible-agent (worker): region files created ok region = 006f31eb-3867-4dd4-8c81-c85062376609
792026-06-25T01:13:22.262ZINFOcrucible-agent (datafile): region 006f31eb-3867-4dd4-8c81-c85062376609 state: Requested -> Created
802026-06-25T01:13:22.262ZINFOcrucible-agent (worker): applying SMF actions post create...
812026-06-25T01:13:22.264ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-006f31eb-3867-4dd4-8c81-c85062376609
822026-06-25T01:13:22.275ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-006f31eb-3867-4dd4-8c81-c85062376609
832026-06-25T01:13:22.280ZINFOcrucible-agent (worker): creating config property group
842026-06-25T01:13:22.282ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-006f31eb-3867-4dd4-8c81-c85062376609
852026-06-25T01:13:22.282ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/006f31eb-3867-4dd4-8c81-c85062376609
862026-06-25T01:13:22.282ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
872026-06-25T01:13:22.282ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
882026-06-25T01:13:22.282ZINFOcrucible-agent (worker): commit
892026-06-25T01:13:22.284ZINFOcrucible-agent (worker): ok!
902026-06-25T01:13:22.289ZINFOcrucible-agent (worker): SMF ok!
912026-06-25T01:13:22.324ZINFOcrucible-agent (dropshot): request completed latency_us = 80 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:59559 req_id = 60bce1ea-bced-471b-bcad-6b70f7e0a537 response_code = 200 uri = /crucible/0/regions
922026-06-25T01:13:46.844ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:59559 req_id = ca05e213-c17a-47e0-a6db-a6ebb67f0f7d response_code = 200 uri = /crucible/0/regions/006f31eb-3867-4dd4-8c81-c85062376609
932026-06-25T01:13:46.848ZINFOcrucible-agent (datafile): path is "/data/regions/006f31eb-3867-4dd4-8c81-c85062376609"
942026-06-25T01:13:46.859ZINFOcrucible-agent (datafile): dataset is oxp_950e579c-96c4-4dc8-bfb1-eef131a5818b/crucible/regions/006f31eb-3867-4dd4-8c81-c85062376609
952026-06-25T01:13:46.866ZINFOcrucible-agent (datafile): region 006f31eb-3867-4dd4-8c81-c85062376609 state: Created -> Tombstoned
962026-06-25T01:13:46.866ZINFOcrucible-agent (worker): applying SMF actions before removal...
972026-06-25T01:13:46.866ZINFOcrucible-agent (dropshot): request completed latency_us = 18235 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:59559 req_id = bae06ca0-e1e7-4f17-b8bc-c15e6d6029e9 response_code = 204 uri = /crucible/0/regions/006f31eb-3867-4dd4-8c81-c85062376609
982026-06-25T01:13:46.868ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-006f31eb-3867-4dd4-8c81-c85062376609 (instance states: (Some(Online), None))
992026-06-25T01:13:46.870ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:59559 req_id = 6406472e-18c6-44fe-ada8-cbbcd54480d3 response_code = 200 uri = /crucible/0/regions/006f31eb-3867-4dd4-8c81-c85062376609
1002026-06-25T01:13:46.872ZINFOcrucible-agent (worker): SMF ok!
1012026-06-25T01:13:46.880ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_950e579c-96c4-4dc8-bfb1-eef131a5818b/crucible/regions/006f31eb-3867-4dd4-8c81-c85062376609" region = 006f31eb-3867-4dd4-8c81-c85062376609
1022026-06-25T01:13:46.929ZINFOcrucible-agent (datafile): region 006f31eb-3867-4dd4-8c81-c85062376609 state: Tombstoned -> Destroyed
1032026-06-25T01:13:46.955ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:59559 req_id = 9371d8c2-ba58-451c-8c34-fe8d002d9687 response_code = 200 uri = /crucible/0/regions/006f31eb-3867-4dd4-8c81-c85062376609