Max level shown:
12024-09-09T21:01:09.938ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:59105
22024-09-09T21:01:09.968ZINFOcrucible-agent (datafile): region ffa35551-d867-4ea4-ac7f-f28cfa0363e2 state: Requested
32024-09-09T21:01:09.973ZINFOcrucible-agent (dropshot): request completed latency_us = 20770 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:59105 req_id = c1e3243c-ad78-486f-a762-452eccb6760a response_code = 200 uri = /crucible/0/regions
42024-09-09T21:01:09.978ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
52024-09-09T21:01:09.988ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_3634a2c6-2441-44d1-961c-492b8f9a328b/crucible/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2
62024-09-09T21:01:09.988ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_3634a2c6-2441-44d1-961c-492b8f9a328b/crucible/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2
72024-09-09T21:01:10.054ZINFOcrucible-agent (worker): creating region Region { id: RegionId("ffa35551-d867-4ea4-ac7f-f28cfa0363e2"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2" region = ffa35551-d867-4ea4-ac7f-f28cfa0363e2
82024-09-09T21:01:10.056ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "ffa35551-d867-4ea4-ac7f-f28cfa0363e2" "--data" "/data/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = ffa35551-d867-4ea4-ac7f-f28cfa0363e2
92024-09-09T21:01:10.194ZINFOcrucible-agent (worker): region files created ok region = ffa35551-d867-4ea4-ac7f-f28cfa0363e2
102024-09-09T21:01:10.194ZINFOcrucible-agent (datafile): region ffa35551-d867-4ea4-ac7f-f28cfa0363e2 state: Requested -> Created
112024-09-09T21:01:10.194ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-09-09T21:01:10.195ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-ffa35551-d867-4ea4-ac7f-f28cfa0363e2
132024-09-09T21:01:10.201ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-ffa35551-d867-4ea4-ac7f-f28cfa0363e2
142024-09-09T21:01:10.220ZINFOcrucible-agent (worker): creating config property group
152024-09-09T21:01:10.223ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ffa35551-d867-4ea4-ac7f-f28cfa0363e2
162024-09-09T21:01:10.234ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2
172024-09-09T21:01:10.236ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
182024-09-09T21:01:10.236ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
192024-09-09T21:01:10.236ZINFOcrucible-agent (worker): commit
202024-09-09T21:01:10.240ZINFOcrucible-agent (worker): ok!
212024-09-09T21:01:10.247ZINFOcrucible-agent (worker): SMF ok!
222024-09-09T21:01:10.285ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:59105 req_id = a41f5789-be29-4334-a948-f99267a261df response_code = 200 uri = /crucible/0/regions
232024-09-09T21:06:27.810ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:53726
242024-09-09T21:06:27.813ZINFOcrucible-agent (dropshot): request completed latency_us = 3172 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:53726 req_id = aae6f27b-c8cd-4fa7-bd1b-7cb39801f688 response_code = 200 uri = /crucible/0/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2
252024-09-09T21:06:27.816ZINFOcrucible-agent (datafile): path is "/data/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2"
262024-09-09T21:06:27.826ZINFOcrucible-agent (datafile): dataset is oxp_3634a2c6-2441-44d1-961c-492b8f9a328b/crucible/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2
272024-09-09T21:06:27.836ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_3634a2c6-2441-44d1-961c-492b8f9a328b/crucible/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2@951fdef0-d0dd-499c-b915-94460e51b7f8
282024-09-09T21:06:27.836ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_3634a2c6-2441-44d1-961c-492b8f9a328b/crucible/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2", "951fdef0-d0dd-499c-b915-94460e51b7f8"]
292024-09-09T21:06:27.848ZINFOcrucible-agent (dropshot): request completed latency_us = 31862 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:53726 req_id = 32d1f7af-139a-48ae-8dce-05bb12a9b74f response_code = 200 uri = /crucible/0/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2/snapshots/951fdef0-d0dd-499c-b915-94460e51b7f8
302024-09-09T21:06:27.851ZINFOcrucible-agent (datafile): path is "/data/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2"
312024-09-09T21:06:27.860ZINFOcrucible-agent (datafile): dataset is oxp_3634a2c6-2441-44d1-961c-492b8f9a328b/crucible/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2
322024-09-09T21:06:27.869ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_3634a2c6-2441-44d1-961c-492b8f9a328b/crucible/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2@951fdef0-d0dd-499c-b915-94460e51b7f8
332024-09-09T21:06:27.869ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_3634a2c6-2441-44d1-961c-492b8f9a328b/crucible/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2", "951fdef0-d0dd-499c-b915-94460e51b7f8"]
342024-09-09T21:06:27.879ZINFOcrucible-agent (datafile): requesting running snapshot ffa35551-d867-4ea4-ac7f-f28cfa0363e2-951fdef0-d0dd-499c-b915-94460e51b7f8 state: Requested
352024-09-09T21:06:27.880ZINFOcrucible-agent (worker): applying SMF actions for region ffa35551-d867-4ea4-ac7f-f28cfa0363e2 running snapshot 951fdef0-d0dd-499c-b915-94460e51b7f8 (state Requested)...
362024-09-09T21:06:27.880ZINFOcrucible-agent (dropshot): request completed latency_us = 28750 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53726 req_id = 0654d7d5-1308-4115-886d-75118f83f7f3 response_code = 200 uri = /crucible/0/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2/snapshots/951fdef0-d0dd-499c-b915-94460e51b7f8/run
372024-09-09T21:06:27.889ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2
382024-09-09T21:06:27.889ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
392024-09-09T21:06:27.889ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
402024-09-09T21:06:27.889ZINFOcrucible-agent (worker): reconfig required
412024-09-09T21:06:27.889ZINFOcrucible-agent (worker): using existing config property group
422024-09-09T21:06:27.889ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ffa35551-d867-4ea4-ac7f-f28cfa0363e2
432024-09-09T21:06:27.889ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2
442024-09-09T21:06:27.891ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
452024-09-09T21:06:27.891ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
462024-09-09T21:06:27.891ZINFOcrucible-agent (worker): commit
472024-09-09T21:06:27.906ZINFOcrucible-agent (worker): ok!
482024-09-09T21:06:27.907ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-ffa35551-d867-4ea4-ac7f-f28cfa0363e2-951fdef0-d0dd-499c-b915-94460e51b7f8
492024-09-09T21:06:27.916ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-ffa35551-d867-4ea4-ac7f-f28cfa0363e2-951fdef0-d0dd-499c-b915-94460e51b7f8
502024-09-09T21:06:27.943ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-ffa35551-d867-4ea4-ac7f-f28cfa0363e2-951fdef0-d0dd-499c-b915-94460e51b7f8
512024-09-09T21:06:27.944ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2/.zfs/snapshot/951fdef0-d0dd-499c-b915-94460e51b7f8
522024-09-09T21:06:27.944ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
532024-09-09T21:06:27.944ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
542024-09-09T21:06:27.944ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
552024-09-09T21:06:27.944ZINFOcrucible-agent (worker): commit
562024-09-09T21:06:27.954ZINFOcrucible-agent (worker): ok!
572024-09-09T21:06:27.966ZINFOcrucible-agent (worker): SMF ok!
582024-09-09T21:06:27.966ZINFOcrucible-agent (datafile): region ffa35551-d867-4ea4-ac7f-f28cfa0363e2 running snapshot 951fdef0-d0dd-499c-b915-94460e51b7f8 state: Requested -> Created
592024-09-09T21:06:28.185ZINFOcrucible-agent (datafile): path is "/data/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2"
602024-09-09T21:06:28.195ZINFOcrucible-agent (datafile): dataset is oxp_3634a2c6-2441-44d1-961c-492b8f9a328b/crucible/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2
612024-09-09T21:06:28.203ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_3634a2c6-2441-44d1-961c-492b8f9a328b/crucible/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2@951fdef0-d0dd-499c-b915-94460e51b7f8
622024-09-09T21:06:28.203ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_3634a2c6-2441-44d1-961c-492b8f9a328b/crucible/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2", "951fdef0-d0dd-499c-b915-94460e51b7f8"]
632024-09-09T21:06:28.211ZINFOcrucible-agent (dropshot): request completed latency_us = 25910 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53726 req_id = 2d7811ab-f453-4d12-bb56-b4b24950d819 response_code = 200 uri = /crucible/0/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2/snapshots/951fdef0-d0dd-499c-b915-94460e51b7f8/run
642024-09-09T21:06:31.738ZINFOcrucible-agent (datafile): region 3344b611-208b-4ce3-a166-11e1a019851a state: Requested
652024-09-09T21:06:31.739ZINFOcrucible-agent (dropshot): request completed latency_us = 226 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53726 req_id = 2f71746e-9a6b-4e62-8555-a04b06c5b290 response_code = 200 uri = /crucible/0/regions
662024-09-09T21:06:31.739ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
672024-09-09T21:06:31.747ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_3634a2c6-2441-44d1-961c-492b8f9a328b/crucible/regions/3344b611-208b-4ce3-a166-11e1a019851a
682024-09-09T21:06:31.747ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_3634a2c6-2441-44d1-961c-492b8f9a328b/crucible/regions/3344b611-208b-4ce3-a166-11e1a019851a
692024-09-09T21:06:31.809ZINFOcrucible-agent (worker): creating region Region { id: RegionId("3344b611-208b-4ce3-a166-11e1a019851a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/3344b611-208b-4ce3-a166-11e1a019851a" region = 3344b611-208b-4ce3-a166-11e1a019851a
702024-09-09T21:06:31.809ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "3344b611-208b-4ce3-a166-11e1a019851a" "--data" "/data/regions/3344b611-208b-4ce3-a166-11e1a019851a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 3344b611-208b-4ce3-a166-11e1a019851a
712024-09-09T21:06:31.950ZINFOcrucible-agent (worker): region files created ok region = 3344b611-208b-4ce3-a166-11e1a019851a
722024-09-09T21:06:31.950ZINFOcrucible-agent (datafile): region 3344b611-208b-4ce3-a166-11e1a019851a state: Requested -> Created
732024-09-09T21:06:31.950ZINFOcrucible-agent (worker): applying SMF actions post create...
742024-09-09T21:06:31.951ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-3344b611-208b-4ce3-a166-11e1a019851a
752024-09-09T21:06:31.957ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-3344b611-208b-4ce3-a166-11e1a019851a
762024-09-09T21:06:31.967ZINFOcrucible-agent (worker): creating config property group
772024-09-09T21:06:31.970ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3344b611-208b-4ce3-a166-11e1a019851a
782024-09-09T21:06:31.970ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3344b611-208b-4ce3-a166-11e1a019851a
792024-09-09T21:06:31.970ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
802024-09-09T21:06:31.970ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
812024-09-09T21:06:31.970ZINFOcrucible-agent (worker): commit
822024-09-09T21:06:31.975ZINFOcrucible-agent (worker): ok!
832024-09-09T21:06:31.980ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2
842024-09-09T21:06:31.980ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
852024-09-09T21:06:31.980ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
862024-09-09T21:06:31.980ZINFOcrucible-agent (worker): reconfig required
872024-09-09T21:06:31.980ZINFOcrucible-agent (worker): using existing config property group
882024-09-09T21:06:31.980ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ffa35551-d867-4ea4-ac7f-f28cfa0363e2
892024-09-09T21:06:31.985ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2
902024-09-09T21:06:31.985ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
912024-09-09T21:06:31.985ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
922024-09-09T21:06:31.985ZINFOcrucible-agent (worker): commit
932024-09-09T21:06:31.989ZINFOcrucible-agent (worker): ok!
942024-09-09T21:06:31.991ZINFOcrucible-agent (worker): SMF ok!
952024-09-09T21:06:32.091ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53726 req_id = 4d070780-ad4b-46d6-a561-5063717ce5a5 response_code = 200 uri = /crucible/0/regions
962024-09-09T21:06:57.284ZINFOcrucible-agent (dropshot): request completed latency_us = 50 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:53726 req_id = 1944a67b-a0f8-4864-9a50-9e3e2cb09784 response_code = 200 uri = /crucible/0/regions/3344b611-208b-4ce3-a166-11e1a019851a
972024-09-09T21:06:57.286ZINFOcrucible-agent (datafile): path is "/data/regions/3344b611-208b-4ce3-a166-11e1a019851a"
982024-09-09T21:06:57.298ZINFOcrucible-agent (datafile): dataset is oxp_3634a2c6-2441-44d1-961c-492b8f9a328b/crucible/regions/3344b611-208b-4ce3-a166-11e1a019851a
992024-09-09T21:06:57.306ZINFOcrucible-agent (datafile): region 3344b611-208b-4ce3-a166-11e1a019851a state: Created -> Tombstoned
1002024-09-09T21:06:57.306ZINFOcrucible-agent (worker): applying SMF actions before removal...
1012024-09-09T21:06:57.306ZINFOcrucible-agent (dropshot): request completed latency_us = 19937 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:53726 req_id = a70a3532-af6d-40bd-916e-c24b81c2c7ca response_code = 204 uri = /crucible/0/regions/3344b611-208b-4ce3-a166-11e1a019851a
1022024-09-09T21:06:57.307ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-3344b611-208b-4ce3-a166-11e1a019851a (instance states: (Some(Online), None))
1032024-09-09T21:06:57.309ZINFOcrucible-agent (dropshot): request completed latency_us = 23 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:53726 req_id = db782953-cde4-49f8-87c4-e0dddbace0cb response_code = 200 uri = /crucible/0/regions/3344b611-208b-4ce3-a166-11e1a019851a
1042024-09-09T21:06:57.313ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2
1052024-09-09T21:06:57.313ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1062024-09-09T21:06:57.313ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
1072024-09-09T21:06:57.313ZINFOcrucible-agent (worker): reconfig required
1082024-09-09T21:06:57.313ZINFOcrucible-agent (worker): using existing config property group
1092024-09-09T21:06:57.314ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ffa35551-d867-4ea4-ac7f-f28cfa0363e2
1102024-09-09T21:06:57.314ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ffa35551-d867-4ea4-ac7f-f28cfa0363e2
1112024-09-09T21:06:57.314ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1122024-09-09T21:06:57.314ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
1132024-09-09T21:06:57.314ZINFOcrucible-agent (worker): commit
1142024-09-09T21:06:57.318ZINFOcrucible-agent (worker): ok!
1152024-09-09T21:06:57.319ZINFOcrucible-agent (worker): SMF ok!
1162024-09-09T21:06:57.328ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_3634a2c6-2441-44d1-961c-492b8f9a328b/crucible/regions/3344b611-208b-4ce3-a166-11e1a019851a" region = 3344b611-208b-4ce3-a166-11e1a019851a
1172024-09-09T21:06:57.403ZINFOcrucible-agent (datafile): region 3344b611-208b-4ce3-a166-11e1a019851a state: Tombstoned -> Destroyed
1182024-09-09T21:06:57.450ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:53726 req_id = 6f4de59b-cc65-4c0a-93cf-1600f69c329e response_code = 200 uri = /crucible/0/regions/3344b611-208b-4ce3-a166-11e1a019851a