Max level shown:
12024-09-03T21:31:25.515ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:64281
22024-09-03T21:31:25.547ZINFOcrucible-agent (datafile): region 12e68e5b-0461-4b91-90f4-f1af578f0b1c state: Requested
32024-09-03T21:31:25.552ZINFOcrucible-agent (dropshot): request completed latency_us = 22752 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64281 req_id = 46f0cd60-6f1f-4056-8e4a-938ea1f317c7 response_code = 200 uri = /crucible/0/regions
42024-09-03T21:31:25.556ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
52024-09-03T21:31:25.568ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_f936d8c3-1ac2-4fd5-84ed-415a39983d55/crucible/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c
62024-09-03T21:31:25.568ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_f936d8c3-1ac2-4fd5-84ed-415a39983d55/crucible/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c
72024-09-03T21:31:25.614ZINFOcrucible-agent (worker): creating region Region { id: RegionId("12e68e5b-0461-4b91-90f4-f1af578f0b1c"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c" region = 12e68e5b-0461-4b91-90f4-f1af578f0b1c
82024-09-03T21:31:25.617ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "12e68e5b-0461-4b91-90f4-f1af578f0b1c" "--data" "/data/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 12e68e5b-0461-4b91-90f4-f1af578f0b1c
92024-09-03T21:31:25.859ZINFOcrucible-agent (worker): region files created ok region = 12e68e5b-0461-4b91-90f4-f1af578f0b1c
102024-09-03T21:31:25.859ZINFOcrucible-agent (datafile): region 12e68e5b-0461-4b91-90f4-f1af578f0b1c state: Requested -> Created
112024-09-03T21:31:25.860ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-09-03T21:31:25.862ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-12e68e5b-0461-4b91-90f4-f1af578f0b1c
132024-09-03T21:31:25.867ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-12e68e5b-0461-4b91-90f4-f1af578f0b1c
142024-09-03T21:31:25.884ZINFOcrucible-agent (worker): creating config property group
152024-09-03T21:31:25.887ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-12e68e5b-0461-4b91-90f4-f1af578f0b1c
162024-09-03T21:31:25.896ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64281 req_id = f426f700-02ce-48db-b49f-6c15e9ba051d response_code = 200 uri = /crucible/0/regions
172024-09-03T21:31:25.898ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c
182024-09-03T21:31:25.900ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192024-09-03T21:31:25.900ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
202024-09-03T21:31:25.900ZINFOcrucible-agent (worker): commit
212024-09-03T21:31:25.904ZINFOcrucible-agent (worker): ok!
222024-09-03T21:31:25.910ZINFOcrucible-agent (worker): SMF ok!
232024-09-03T21:36:41.489ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:33152
242024-09-03T21:36:41.489ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:33152 req_id = ec4a3102-d6f9-4d36-95f9-cd233e4f086b response_code = 200 uri = /crucible/0/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c
252024-09-03T21:36:41.493ZINFOcrucible-agent (datafile): path is "/data/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c"
262024-09-03T21:36:41.502ZINFOcrucible-agent (datafile): dataset is oxp_f936d8c3-1ac2-4fd5-84ed-415a39983d55/crucible/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c
272024-09-03T21:36:41.510ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_f936d8c3-1ac2-4fd5-84ed-415a39983d55/crucible/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c@45c10264-cac8-4da8-85cc-3ec557aec3bd
282024-09-03T21:36:41.510ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_f936d8c3-1ac2-4fd5-84ed-415a39983d55/crucible/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c", "45c10264-cac8-4da8-85cc-3ec557aec3bd"]
292024-09-03T21:36:41.522ZINFOcrucible-agent (dropshot): request completed latency_us = 29688 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:33152 req_id = 6c6956c8-9f50-4b49-ac78-afb73c641c50 response_code = 200 uri = /crucible/0/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c/snapshots/45c10264-cac8-4da8-85cc-3ec557aec3bd
302024-09-03T21:36:41.526ZINFOcrucible-agent (datafile): path is "/data/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c"
312024-09-03T21:36:41.534ZINFOcrucible-agent (datafile): dataset is oxp_f936d8c3-1ac2-4fd5-84ed-415a39983d55/crucible/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c
322024-09-03T21:36:41.541ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_f936d8c3-1ac2-4fd5-84ed-415a39983d55/crucible/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c@45c10264-cac8-4da8-85cc-3ec557aec3bd
332024-09-03T21:36:41.541ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_f936d8c3-1ac2-4fd5-84ed-415a39983d55/crucible/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c", "45c10264-cac8-4da8-85cc-3ec557aec3bd"]
342024-09-03T21:36:41.549ZINFOcrucible-agent (datafile): requesting running snapshot 12e68e5b-0461-4b91-90f4-f1af578f0b1c-45c10264-cac8-4da8-85cc-3ec557aec3bd state: Requested
352024-09-03T21:36:41.550ZINFOcrucible-agent (worker): applying SMF actions for region 12e68e5b-0461-4b91-90f4-f1af578f0b1c running snapshot 45c10264-cac8-4da8-85cc-3ec557aec3bd (state Requested)...
362024-09-03T21:36:41.550ZINFOcrucible-agent (dropshot): request completed latency_us = 24056 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33152 req_id = ada0523c-b4f1-4815-8349-3c3a0ede2ed7 response_code = 200 uri = /crucible/0/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c/snapshots/45c10264-cac8-4da8-85cc-3ec557aec3bd/run
372024-09-03T21:36:41.561ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c
382024-09-03T21:36:41.561ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
392024-09-03T21:36:41.561ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
402024-09-03T21:36:41.561ZINFOcrucible-agent (worker): reconfig required
412024-09-03T21:36:41.561ZINFOcrucible-agent (worker): using existing config property group
422024-09-03T21:36:41.561ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-12e68e5b-0461-4b91-90f4-f1af578f0b1c
432024-09-03T21:36:41.561ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c
442024-09-03T21:36:41.563ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
452024-09-03T21:36:41.563ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
462024-09-03T21:36:41.564ZINFOcrucible-agent (worker): commit
472024-09-03T21:36:41.571ZINFOcrucible-agent (worker): ok!
482024-09-03T21:36:41.572ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-12e68e5b-0461-4b91-90f4-f1af578f0b1c-45c10264-cac8-4da8-85cc-3ec557aec3bd
492024-09-03T21:36:41.580ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-12e68e5b-0461-4b91-90f4-f1af578f0b1c-45c10264-cac8-4da8-85cc-3ec557aec3bd
502024-09-03T21:36:41.598ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-12e68e5b-0461-4b91-90f4-f1af578f0b1c-45c10264-cac8-4da8-85cc-3ec557aec3bd
512024-09-03T21:36:41.599ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c/.zfs/snapshot/45c10264-cac8-4da8-85cc-3ec557aec3bd
522024-09-03T21:36:41.599ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
532024-09-03T21:36:41.599ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
542024-09-03T21:36:41.599ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
552024-09-03T21:36:41.599ZINFOcrucible-agent (worker): commit
562024-09-03T21:36:41.606ZINFOcrucible-agent (worker): ok!
572024-09-03T21:36:41.612ZINFOcrucible-agent (worker): SMF ok!
582024-09-03T21:36:41.612ZINFOcrucible-agent (datafile): region 12e68e5b-0461-4b91-90f4-f1af578f0b1c running snapshot 45c10264-cac8-4da8-85cc-3ec557aec3bd state: Requested -> Created
592024-09-03T21:36:41.915ZINFOcrucible-agent (datafile): path is "/data/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c"
602024-09-03T21:36:41.924ZINFOcrucible-agent (datafile): dataset is oxp_f936d8c3-1ac2-4fd5-84ed-415a39983d55/crucible/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c
612024-09-03T21:36:41.932ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_f936d8c3-1ac2-4fd5-84ed-415a39983d55/crucible/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c@45c10264-cac8-4da8-85cc-3ec557aec3bd
622024-09-03T21:36:41.932ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_f936d8c3-1ac2-4fd5-84ed-415a39983d55/crucible/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c", "45c10264-cac8-4da8-85cc-3ec557aec3bd"]
632024-09-03T21:36:41.939ZINFOcrucible-agent (dropshot): request completed latency_us = 24184 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33152 req_id = d68ae902-5289-480d-a276-1e9c1a900b31 response_code = 200 uri = /crucible/0/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c/snapshots/45c10264-cac8-4da8-85cc-3ec557aec3bd/run
642024-09-03T21:36:45.386ZINFOcrucible-agent (datafile): region de91351c-9f1d-4aa8-8af2-709b0bb49a8f state: Requested
652024-09-03T21:36:45.386ZINFOcrucible-agent (dropshot): request completed latency_us = 212 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33152 req_id = e43ac412-6b5e-4a84-9b7b-c1e7d02f4d87 response_code = 200 uri = /crucible/0/regions
662024-09-03T21:36:45.386ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
672024-09-03T21:36:45.395ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_f936d8c3-1ac2-4fd5-84ed-415a39983d55/crucible/regions/de91351c-9f1d-4aa8-8af2-709b0bb49a8f
682024-09-03T21:36:45.396ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_f936d8c3-1ac2-4fd5-84ed-415a39983d55/crucible/regions/de91351c-9f1d-4aa8-8af2-709b0bb49a8f
692024-09-03T21:36:45.453ZINFOcrucible-agent (worker): creating region Region { id: RegionId("de91351c-9f1d-4aa8-8af2-709b0bb49a8f"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/de91351c-9f1d-4aa8-8af2-709b0bb49a8f" region = de91351c-9f1d-4aa8-8af2-709b0bb49a8f
702024-09-03T21:36:45.453ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "de91351c-9f1d-4aa8-8af2-709b0bb49a8f" "--data" "/data/regions/de91351c-9f1d-4aa8-8af2-709b0bb49a8f" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = de91351c-9f1d-4aa8-8af2-709b0bb49a8f
712024-09-03T21:36:45.533ZINFOcrucible-agent (worker): region files created ok region = de91351c-9f1d-4aa8-8af2-709b0bb49a8f
722024-09-03T21:36:45.533ZINFOcrucible-agent (datafile): region de91351c-9f1d-4aa8-8af2-709b0bb49a8f state: Requested -> Created
732024-09-03T21:36:45.534ZINFOcrucible-agent (worker): applying SMF actions post create...
742024-09-03T21:36:45.534ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c
752024-09-03T21:36:45.534ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
762024-09-03T21:36:45.534ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
772024-09-03T21:36:45.534ZINFOcrucible-agent (worker): reconfig required
782024-09-03T21:36:45.534ZINFOcrucible-agent (worker): using existing config property group
792024-09-03T21:36:45.535ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-12e68e5b-0461-4b91-90f4-f1af578f0b1c
802024-09-03T21:36:45.535ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c
812024-09-03T21:36:45.535ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
822024-09-03T21:36:45.535ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
832024-09-03T21:36:45.535ZINFOcrucible-agent (worker): commit
842024-09-03T21:36:45.540ZINFOcrucible-agent (worker): ok!
852024-09-03T21:36:45.540ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-de91351c-9f1d-4aa8-8af2-709b0bb49a8f
862024-09-03T21:36:45.547ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-de91351c-9f1d-4aa8-8af2-709b0bb49a8f
872024-09-03T21:36:45.554ZINFOcrucible-agent (worker): creating config property group
882024-09-03T21:36:45.557ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-de91351c-9f1d-4aa8-8af2-709b0bb49a8f
892024-09-03T21:36:45.557ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/de91351c-9f1d-4aa8-8af2-709b0bb49a8f
902024-09-03T21:36:45.557ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
912024-09-03T21:36:45.558ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
922024-09-03T21:36:45.558ZINFOcrucible-agent (worker): commit
932024-09-03T21:36:45.561ZINFOcrucible-agent (worker): ok!
942024-09-03T21:36:45.567ZINFOcrucible-agent (worker): SMF ok!
952024-09-03T21:36:45.726ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33152 req_id = c65326e0-ab51-45aa-9a4b-8e7a510ecdbe response_code = 200 uri = /crucible/0/regions
962024-09-03T21:37:10.946ZINFOcrucible-agent (dropshot): request completed latency_us = 49 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:33152 req_id = b08b4b26-4971-44bd-bc5d-4548a35b3749 response_code = 200 uri = /crucible/0/regions/de91351c-9f1d-4aa8-8af2-709b0bb49a8f
972024-09-03T21:37:10.949ZINFOcrucible-agent (datafile): path is "/data/regions/de91351c-9f1d-4aa8-8af2-709b0bb49a8f"
982024-09-03T21:37:10.959ZINFOcrucible-agent (datafile): dataset is oxp_f936d8c3-1ac2-4fd5-84ed-415a39983d55/crucible/regions/de91351c-9f1d-4aa8-8af2-709b0bb49a8f
992024-09-03T21:37:10.967ZINFOcrucible-agent (datafile): region de91351c-9f1d-4aa8-8af2-709b0bb49a8f state: Created -> Tombstoned
1002024-09-03T21:37:10.967ZINFOcrucible-agent (worker): applying SMF actions before removal...
1012024-09-03T21:37:10.968ZINFOcrucible-agent (dropshot): request completed latency_us = 19037 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:33152 req_id = 6fd41798-898b-42a1-8cfd-f80e2c8e447f response_code = 204 uri = /crucible/0/regions/de91351c-9f1d-4aa8-8af2-709b0bb49a8f
1022024-09-03T21:37:10.969ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-de91351c-9f1d-4aa8-8af2-709b0bb49a8f (instance states: (Some(Online), None))
1032024-09-03T21:37:10.970ZINFOcrucible-agent (dropshot): request completed latency_us = 25 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:33152 req_id = eb2ef9df-29b7-4aaa-8017-a32e84a51e51 response_code = 200 uri = /crucible/0/regions/de91351c-9f1d-4aa8-8af2-709b0bb49a8f
1042024-09-03T21:37:10.973ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c
1052024-09-03T21:37:10.973ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1062024-09-03T21:37:10.974ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
1072024-09-03T21:37:10.974ZINFOcrucible-agent (worker): reconfig required
1082024-09-03T21:37:10.974ZINFOcrucible-agent (worker): using existing config property group
1092024-09-03T21:37:10.974ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-12e68e5b-0461-4b91-90f4-f1af578f0b1c
1102024-09-03T21:37:10.974ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/12e68e5b-0461-4b91-90f4-f1af578f0b1c
1112024-09-03T21:37:10.974ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1122024-09-03T21:37:10.974ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
1132024-09-03T21:37:10.974ZINFOcrucible-agent (worker): commit
1142024-09-03T21:37:10.978ZINFOcrucible-agent (worker): ok!
1152024-09-03T21:37:10.979ZINFOcrucible-agent (worker): SMF ok!
1162024-09-03T21:37:10.989ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_f936d8c3-1ac2-4fd5-84ed-415a39983d55/crucible/regions/de91351c-9f1d-4aa8-8af2-709b0bb49a8f" region = de91351c-9f1d-4aa8-8af2-709b0bb49a8f
1172024-09-03T21:37:11.037ZINFOcrucible-agent (datafile): region de91351c-9f1d-4aa8-8af2-709b0bb49a8f state: Tombstoned -> Destroyed
1182024-09-03T21:37:11.046ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:33152 req_id = 149a9e65-6eb7-4d3b-ae2a-d41f578eafe5 response_code = 200 uri = /crucible/0/regions/de91351c-9f1d-4aa8-8af2-709b0bb49a8f