Max level shown:
1 [ Mar 25 22:49:21 Enabled. ]
2 [ Mar 25 22:49:21 Rereading configuration. ]
3 [ Mar 25 22:49:25 Rereading configuration. ]
4 [ Mar 25 22:49:33 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible -l [fd00:1122:3344:101::15]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-03-25T22:49:33.409ZINFOcrucible-agent: dataset: "oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible"
72025-03-25T22:49:33.413ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::15]:32345
82025-03-25T22:49:33.413ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92025-03-25T22:49:33.432ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102025-03-25T22:49:33.499ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::15]:32345
112025-03-25T22:49:52.405ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::c]:42316
122025-03-25T22:49:52.439ZINFOcrucible-agent (datafile): region 313b7a16-eb52-4d1c-bba1-0f54bcecc26b state: Requested
132025-03-25T22:49:52.440ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
142025-03-25T22:49:52.445ZINFOcrucible-agent (dropshot): request completed latency_us = 18950 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:42316 req_id = 4a581192-3624-49c4-ba19-d3c0bef8c586 response_code = 200 uri = /crucible/0/regions
152025-03-25T22:49:52.453ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b
162025-03-25T22:49:52.454ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b
172025-03-25T22:49:52.553ZINFOcrucible-agent (worker): creating region Region { id: RegionId("313b7a16-eb52-4d1c-bba1-0f54bcecc26b"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b" region = 313b7a16-eb52-4d1c-bba1-0f54bcecc26b
182025-03-25T22:49:52.554ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "313b7a16-eb52-4d1c-bba1-0f54bcecc26b" "--data" "/data/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 313b7a16-eb52-4d1c-bba1-0f54bcecc26b
192025-03-25T22:49:52.723ZINFOcrucible-agent (dropshot): request completed latency_us = 87 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:42316 req_id = 8dbe1ca6-025e-4875-b233-ce124707e3c7 response_code = 200 uri = /crucible/0/regions
202025-03-25T22:49:52.741ZINFOcrucible-agent (worker): region files created ok region = 313b7a16-eb52-4d1c-bba1-0f54bcecc26b
212025-03-25T22:49:52.741ZINFOcrucible-agent (datafile): region 313b7a16-eb52-4d1c-bba1-0f54bcecc26b state: Requested -> Created
222025-03-25T22:49:52.741ZINFOcrucible-agent (worker): applying SMF actions post create...
232025-03-25T22:49:52.741ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-313b7a16-eb52-4d1c-bba1-0f54bcecc26b
242025-03-25T22:49:52.747ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-313b7a16-eb52-4d1c-bba1-0f54bcecc26b
252025-03-25T22:49:52.761ZINFOcrucible-agent (worker): creating config property group
262025-03-25T22:49:52.766ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-313b7a16-eb52-4d1c-bba1-0f54bcecc26b
272025-03-25T22:49:52.766ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b
282025-03-25T22:49:52.766ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292025-03-25T22:49:52.767ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
302025-03-25T22:49:52.767ZINFOcrucible-agent (worker): commit
312025-03-25T22:49:52.776ZINFOcrucible-agent (worker): ok!
322025-03-25T22:49:52.792ZINFOcrucible-agent (worker): SMF ok!
332025-03-25T22:49:53.224ZINFOcrucible-agent (dropshot): request completed latency_us = 46 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:42316 req_id = c65dd9f3-f4c3-4b01-ad08-308a0d2b926d response_code = 200 uri = /crucible/0/regions
342025-03-25T22:59:53.560ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::c]:33135
352025-03-25T22:59:53.560ZINFOcrucible-agent (datafile): region 791f641c-4ab6-4c3d-a231-9a20ec7a40dd state: Requested
362025-03-25T22:59:53.561ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
372025-03-25T22:59:53.561ZINFOcrucible-agent (dropshot): request completed latency_us = 635 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:33135 req_id = 16369e8a-6f32-44fe-96aa-a59cd016ddb6 response_code = 200 uri = /crucible/0/regions
382025-03-25T22:59:53.573ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible/regions/791f641c-4ab6-4c3d-a231-9a20ec7a40dd
392025-03-25T22:59:53.573ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible/regions/791f641c-4ab6-4c3d-a231-9a20ec7a40dd
402025-03-25T22:59:53.622ZINFOcrucible-agent (worker): creating region Region { id: RegionId("791f641c-4ab6-4c3d-a231-9a20ec7a40dd"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/791f641c-4ab6-4c3d-a231-9a20ec7a40dd" region = 791f641c-4ab6-4c3d-a231-9a20ec7a40dd
412025-03-25T22:59:53.624ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "791f641c-4ab6-4c3d-a231-9a20ec7a40dd" "--data" "/data/regions/791f641c-4ab6-4c3d-a231-9a20ec7a40dd" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 791f641c-4ab6-4c3d-a231-9a20ec7a40dd
422025-03-25T22:59:53.751ZINFOcrucible-agent (worker): region files created ok region = 791f641c-4ab6-4c3d-a231-9a20ec7a40dd
432025-03-25T22:59:53.751ZINFOcrucible-agent (datafile): region 791f641c-4ab6-4c3d-a231-9a20ec7a40dd state: Requested -> Created
442025-03-25T22:59:53.751ZINFOcrucible-agent (worker): applying SMF actions post create...
452025-03-25T22:59:53.761ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-791f641c-4ab6-4c3d-a231-9a20ec7a40dd
462025-03-25T22:59:53.765ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-791f641c-4ab6-4c3d-a231-9a20ec7a40dd
472025-03-25T22:59:53.774ZINFOcrucible-agent (dropshot): request completed latency_us = 1224 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:33135 req_id = fe64e177-d6c0-445c-8f65-8d3e8872216c response_code = 200 uri = /crucible/0/regions
482025-03-25T22:59:53.775ZINFOcrucible-agent (worker): creating config property group
492025-03-25T22:59:53.779ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-791f641c-4ab6-4c3d-a231-9a20ec7a40dd
502025-03-25T22:59:53.780ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/791f641c-4ab6-4c3d-a231-9a20ec7a40dd
512025-03-25T22:59:53.780ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
522025-03-25T22:59:53.780ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
532025-03-25T22:59:53.780ZINFOcrucible-agent (worker): commit
542025-03-25T22:59:53.784ZINFOcrucible-agent (worker): ok!
552025-03-25T22:59:53.788ZINFOcrucible-agent (worker): SMF ok!
562025-03-25T22:59:54.793ZINFOcrucible-agent (dropshot): request completed latency_us = 2882 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:33135 req_id = 816b7180-8071-4109-a3ec-5aae8d16cf91 response_code = 200 uri = /crucible/0/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b
572025-03-25T22:59:54.798ZINFOcrucible-agent (datafile): path is "/data/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b"
582025-03-25T22:59:54.808ZINFOcrucible-agent (datafile): dataset is oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b
592025-03-25T22:59:54.817ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b@2622e9be-a829-465b-8f08-acde5d239349
602025-03-25T22:59:54.817ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b", "2622e9be-a829-465b-8f08-acde5d239349"]
612025-03-25T22:59:54.827ZINFOcrucible-agent (dropshot): request completed latency_us = 31710 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:33135 req_id = 9d5a988f-cdfa-413b-83c7-693e17bf7568 response_code = 200 uri = /crucible/0/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b/snapshots/2622e9be-a829-465b-8f08-acde5d239349
622025-03-25T22:59:54.831ZINFOcrucible-agent (datafile): path is "/data/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b"
632025-03-25T22:59:54.839ZINFOcrucible-agent (datafile): dataset is oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b
642025-03-25T22:59:54.849ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b@2622e9be-a829-465b-8f08-acde5d239349
652025-03-25T22:59:54.849ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b", "2622e9be-a829-465b-8f08-acde5d239349"]
662025-03-25T22:59:54.858ZINFOcrucible-agent (datafile): requesting running snapshot 313b7a16-eb52-4d1c-bba1-0f54bcecc26b-2622e9be-a829-465b-8f08-acde5d239349 state: Requested
672025-03-25T22:59:54.858ZINFOcrucible-agent (worker): applying SMF actions for region 313b7a16-eb52-4d1c-bba1-0f54bcecc26b running snapshot 2622e9be-a829-465b-8f08-acde5d239349 (state Requested)...
682025-03-25T22:59:54.858ZINFOcrucible-agent (dropshot): request completed latency_us = 26986 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:33135 req_id = 234bf442-bb78-4c5f-b66e-73be4a335edc response_code = 200 uri = /crucible/0/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b/snapshots/2622e9be-a829-465b-8f08-acde5d239349/run
692025-03-25T22:59:54.860ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-313b7a16-eb52-4d1c-bba1-0f54bcecc26b-2622e9be-a829-465b-8f08-acde5d239349
702025-03-25T22:59:54.871ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-313b7a16-eb52-4d1c-bba1-0f54bcecc26b-2622e9be-a829-465b-8f08-acde5d239349
712025-03-25T22:59:54.881ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-313b7a16-eb52-4d1c-bba1-0f54bcecc26b-2622e9be-a829-465b-8f08-acde5d239349
722025-03-25T22:59:54.881ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b/.zfs/snapshot/2622e9be-a829-465b-8f08-acde5d239349
732025-03-25T22:59:54.881ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
742025-03-25T22:59:54.881ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
752025-03-25T22:59:54.881ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
762025-03-25T22:59:54.881ZINFOcrucible-agent (worker): commit
772025-03-25T22:59:54.886ZINFOcrucible-agent (worker): ok!
782025-03-25T22:59:54.890ZINFOcrucible-agent (worker): SMF ok!
792025-03-25T22:59:54.890ZINFOcrucible-agent (datafile): region 313b7a16-eb52-4d1c-bba1-0f54bcecc26b running snapshot 2622e9be-a829-465b-8f08-acde5d239349 state: Requested -> Created
802025-03-25T22:59:55.095ZINFOcrucible-agent (datafile): path is "/data/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b"
812025-03-25T22:59:55.104ZINFOcrucible-agent (datafile): dataset is oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b
822025-03-25T22:59:55.112ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b@2622e9be-a829-465b-8f08-acde5d239349
832025-03-25T22:59:55.112ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b", "2622e9be-a829-465b-8f08-acde5d239349"]
842025-03-25T22:59:55.120ZINFOcrucible-agent (dropshot): request completed latency_us = 24847 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:33135 req_id = 61bebd7a-21f2-44a2-94d4-ecbc5f1bb1c1 response_code = 200 uri = /crucible/0/regions/313b7a16-eb52-4d1c-bba1-0f54bcecc26b/snapshots/2622e9be-a829-465b-8f08-acde5d239349/run
852025-03-25T22:59:59.681ZINFOcrucible-agent (datafile): region d7dac8a1-c392-4ea2-9d86-9b530e1d8b62 state: Requested
862025-03-25T22:59:59.681ZINFOcrucible-agent (dropshot): request completed latency_us = 256 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:33135 req_id = 1953bc50-3d6b-415f-a87a-6e686a17e6fa response_code = 200 uri = /crucible/0/regions
872025-03-25T22:59:59.681ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
882025-03-25T22:59:59.691ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible/regions/d7dac8a1-c392-4ea2-9d86-9b530e1d8b62
892025-03-25T22:59:59.691ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_d0b95f05-9c6c-4be8-b268-bbafec767714/crucible/regions/d7dac8a1-c392-4ea2-9d86-9b530e1d8b62
902025-03-25T22:59:59.764ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d7dac8a1-c392-4ea2-9d86-9b530e1d8b62"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d7dac8a1-c392-4ea2-9d86-9b530e1d8b62" region = d7dac8a1-c392-4ea2-9d86-9b530e1d8b62
912025-03-25T22:59:59.766ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d7dac8a1-c392-4ea2-9d86-9b530e1d8b62" "--data" "/data/regions/d7dac8a1-c392-4ea2-9d86-9b530e1d8b62" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = d7dac8a1-c392-4ea2-9d86-9b530e1d8b62
922025-03-25T22:59:59.844ZINFOcrucible-agent (worker): region files created ok region = d7dac8a1-c392-4ea2-9d86-9b530e1d8b62
932025-03-25T22:59:59.844ZINFOcrucible-agent (datafile): region d7dac8a1-c392-4ea2-9d86-9b530e1d8b62 state: Requested -> Created
942025-03-25T22:59:59.844ZINFOcrucible-agent (worker): applying SMF actions post create...
952025-03-25T22:59:59.846ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d7dac8a1-c392-4ea2-9d86-9b530e1d8b62
962025-03-25T22:59:59.849ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d7dac8a1-c392-4ea2-9d86-9b530e1d8b62
972025-03-25T22:59:59.856ZINFOcrucible-agent (worker): creating config property group
982025-03-25T22:59:59.859ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d7dac8a1-c392-4ea2-9d86-9b530e1d8b62
992025-03-25T22:59:59.860ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d7dac8a1-c392-4ea2-9d86-9b530e1d8b62
1002025-03-25T22:59:59.860ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
1012025-03-25T22:59:59.860ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
1022025-03-25T22:59:59.860ZINFOcrucible-agent (worker): commit
1032025-03-25T22:59:59.863ZINFOcrucible-agent (worker): ok!
1042025-03-25T22:59:59.869ZINFOcrucible-agent (worker): SMF ok!
1052025-03-25T23:00:00.040ZINFOcrucible-agent (dropshot): request completed latency_us = 117 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:33135 req_id = 80d6641a-e11c-4a0f-b8b0-043c0027d266 response_code = 200 uri = /crucible/0/regions