Max level shown:
12024-12-21T07:15:32.152ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:57741
22024-12-21T07:15:32.190ZINFOcrucible-agent (datafile): region c945c2c5-9fb1-49df-a99c-c51dce8b68af state: Requested
32024-12-21T07:15:32.193ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-12-21T07:15:32.197ZINFOcrucible-agent (dropshot): request completed latency_us = 12468 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57741 req_id = e3092276-0356-4423-ad8c-8fce2fec8743 response_code = 200 uri = /crucible/0/regions
52024-12-21T07:15:32.203ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_66e25da2-039e-41d1-800b-7492e8007ef4/crucible/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af
62024-12-21T07:15:32.203ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_66e25da2-039e-41d1-800b-7492e8007ef4/crucible/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af
72024-12-21T07:15:32.268ZINFOcrucible-agent (worker): creating region Region { id: RegionId("c945c2c5-9fb1-49df-a99c-c51dce8b68af"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af" region = c945c2c5-9fb1-49df-a99c-c51dce8b68af
82024-12-21T07:15:32.271ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "c945c2c5-9fb1-49df-a99c-c51dce8b68af" "--data" "/data/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = c945c2c5-9fb1-49df-a99c-c51dce8b68af
92024-12-21T07:15:32.556ZINFOcrucible-agent (worker): region files created ok region = c945c2c5-9fb1-49df-a99c-c51dce8b68af
102024-12-21T07:15:32.556ZINFOcrucible-agent (datafile): region c945c2c5-9fb1-49df-a99c-c51dce8b68af state: Requested -> Created
112024-12-21T07:15:32.556ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-12-21T07:15:32.557ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-c945c2c5-9fb1-49df-a99c-c51dce8b68af
132024-12-21T07:15:32.562ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-c945c2c5-9fb1-49df-a99c-c51dce8b68af
142024-12-21T07:15:32.567ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57741 req_id = cab16f48-a7e2-4835-a6d2-5e1bf3aa6941 response_code = 200 uri = /crucible/0/regions
152024-12-21T07:15:32.575ZINFOcrucible-agent (worker): creating config property group
162024-12-21T07:15:32.582ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-c945c2c5-9fb1-49df-a99c-c51dce8b68af
172024-12-21T07:15:32.582ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af
182024-12-21T07:15:32.582ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192024-12-21T07:15:32.582ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
202024-12-21T07:15:32.582ZINFOcrucible-agent (worker): commit
212024-12-21T07:15:32.588ZINFOcrucible-agent (worker): ok!
222024-12-21T07:15:32.593ZINFOcrucible-agent (worker): SMF ok!
232024-12-21T07:26:21.959ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:43472
242024-12-21T07:26:21.959ZINFOcrucible-agent (datafile): region e54e075f-b2ae-4bf8-a65a-de40859a88c1 state: Requested
252024-12-21T07:26:21.959ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
262024-12-21T07:26:21.959ZINFOcrucible-agent (dropshot): request completed latency_us = 639 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43472 req_id = 181c6389-22e9-42ce-896d-8f13cdc34a4b response_code = 200 uri = /crucible/0/regions
272024-12-21T07:26:21.971ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_66e25da2-039e-41d1-800b-7492e8007ef4/crucible/regions/e54e075f-b2ae-4bf8-a65a-de40859a88c1
282024-12-21T07:26:21.971ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_66e25da2-039e-41d1-800b-7492e8007ef4/crucible/regions/e54e075f-b2ae-4bf8-a65a-de40859a88c1
292024-12-21T07:26:22.035ZINFOcrucible-agent (worker): creating region Region { id: RegionId("e54e075f-b2ae-4bf8-a65a-de40859a88c1"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/e54e075f-b2ae-4bf8-a65a-de40859a88c1" region = e54e075f-b2ae-4bf8-a65a-de40859a88c1
302024-12-21T07:26:22.035ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "e54e075f-b2ae-4bf8-a65a-de40859a88c1" "--data" "/data/regions/e54e075f-b2ae-4bf8-a65a-de40859a88c1" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = e54e075f-b2ae-4bf8-a65a-de40859a88c1
312024-12-21T07:26:22.112ZINFOcrucible-agent (worker): region files created ok region = e54e075f-b2ae-4bf8-a65a-de40859a88c1
322024-12-21T07:26:22.112ZINFOcrucible-agent (datafile): region e54e075f-b2ae-4bf8-a65a-de40859a88c1 state: Requested -> Created
332024-12-21T07:26:22.112ZINFOcrucible-agent (worker): applying SMF actions post create...
342024-12-21T07:26:22.118ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43472 req_id = e7d5aa68-868a-4b86-be8b-944789f1ad48 response_code = 200 uri = /crucible/0/regions
352024-12-21T07:26:22.122ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-e54e075f-b2ae-4bf8-a65a-de40859a88c1
362024-12-21T07:26:22.127ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-e54e075f-b2ae-4bf8-a65a-de40859a88c1
372024-12-21T07:26:22.138ZINFOcrucible-agent (worker): creating config property group
382024-12-21T07:26:22.142ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-e54e075f-b2ae-4bf8-a65a-de40859a88c1
392024-12-21T07:26:22.142ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e54e075f-b2ae-4bf8-a65a-de40859a88c1
402024-12-21T07:26:22.142ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
412024-12-21T07:26:22.142ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
422024-12-21T07:26:22.142ZINFOcrucible-agent (worker): commit
432024-12-21T07:26:22.145ZINFOcrucible-agent (worker): ok!
442024-12-21T07:26:22.151ZINFOcrucible-agent (worker): SMF ok!
452024-12-21T07:26:23.337ZINFOcrucible-agent (dropshot): request completed latency_us = 2486 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:43472 req_id = 64be44ad-66d4-4c83-9629-710c1e08d9d9 response_code = 200 uri = /crucible/0/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af
462024-12-21T07:26:23.343ZINFOcrucible-agent (datafile): path is "/data/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af"
472024-12-21T07:26:23.353ZINFOcrucible-agent (datafile): dataset is oxp_66e25da2-039e-41d1-800b-7492e8007ef4/crucible/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af
482024-12-21T07:26:23.362ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_66e25da2-039e-41d1-800b-7492e8007ef4/crucible/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af@7f7566db-9859-4847-b89f-c07e9599a3b1
492024-12-21T07:26:23.362ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_66e25da2-039e-41d1-800b-7492e8007ef4/crucible/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af", "7f7566db-9859-4847-b89f-c07e9599a3b1"]
502024-12-21T07:26:23.372ZINFOcrucible-agent (dropshot): request completed latency_us = 29053 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:43472 req_id = 14270601-8747-44f5-8f2d-da2ddffd8d98 response_code = 200 uri = /crucible/0/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af/snapshots/7f7566db-9859-4847-b89f-c07e9599a3b1
512024-12-21T07:26:23.375ZINFOcrucible-agent (datafile): path is "/data/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af"
522024-12-21T07:26:23.384ZINFOcrucible-agent (datafile): dataset is oxp_66e25da2-039e-41d1-800b-7492e8007ef4/crucible/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af
532024-12-21T07:26:23.392ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_66e25da2-039e-41d1-800b-7492e8007ef4/crucible/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af@7f7566db-9859-4847-b89f-c07e9599a3b1
542024-12-21T07:26:23.392ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_66e25da2-039e-41d1-800b-7492e8007ef4/crucible/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af", "7f7566db-9859-4847-b89f-c07e9599a3b1"]
552024-12-21T07:26:23.400ZINFOcrucible-agent (datafile): requesting running snapshot c945c2c5-9fb1-49df-a99c-c51dce8b68af-7f7566db-9859-4847-b89f-c07e9599a3b1 state: Requested
562024-12-21T07:26:23.400ZINFOcrucible-agent (worker): applying SMF actions for region c945c2c5-9fb1-49df-a99c-c51dce8b68af running snapshot 7f7566db-9859-4847-b89f-c07e9599a3b1 (state Requested)...
572024-12-21T07:26:23.400ZINFOcrucible-agent (dropshot): request completed latency_us = 24726 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43472 req_id = 1da06aa8-3549-4de7-8317-7a6308e7581f response_code = 200 uri = /crucible/0/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af/snapshots/7f7566db-9859-4847-b89f-c07e9599a3b1/run
582024-12-21T07:26:23.402ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-c945c2c5-9fb1-49df-a99c-c51dce8b68af-7f7566db-9859-4847-b89f-c07e9599a3b1
592024-12-21T07:26:23.413ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-c945c2c5-9fb1-49df-a99c-c51dce8b68af-7f7566db-9859-4847-b89f-c07e9599a3b1
602024-12-21T07:26:23.424ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-c945c2c5-9fb1-49df-a99c-c51dce8b68af-7f7566db-9859-4847-b89f-c07e9599a3b1
612024-12-21T07:26:23.425ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af/.zfs/snapshot/7f7566db-9859-4847-b89f-c07e9599a3b1
622024-12-21T07:26:23.425ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
632024-12-21T07:26:23.425ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-12-21T07:26:23.425ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
652024-12-21T07:26:23.425ZINFOcrucible-agent (worker): commit
662024-12-21T07:26:23.429ZINFOcrucible-agent (worker): ok!
672024-12-21T07:26:23.434ZINFOcrucible-agent (worker): SMF ok!
682024-12-21T07:26:23.434ZINFOcrucible-agent (datafile): region c945c2c5-9fb1-49df-a99c-c51dce8b68af running snapshot 7f7566db-9859-4847-b89f-c07e9599a3b1 state: Requested -> Created
692024-12-21T07:26:23.742ZINFOcrucible-agent (datafile): path is "/data/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af"
702024-12-21T07:26:23.751ZINFOcrucible-agent (datafile): dataset is oxp_66e25da2-039e-41d1-800b-7492e8007ef4/crucible/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af
712024-12-21T07:26:23.759ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_66e25da2-039e-41d1-800b-7492e8007ef4/crucible/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af@7f7566db-9859-4847-b89f-c07e9599a3b1
722024-12-21T07:26:23.759ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_66e25da2-039e-41d1-800b-7492e8007ef4/crucible/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af", "7f7566db-9859-4847-b89f-c07e9599a3b1"]
732024-12-21T07:26:23.767ZINFOcrucible-agent (dropshot): request completed latency_us = 24488 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43472 req_id = 28021dfd-5461-40fb-a592-938760bcf100 response_code = 200 uri = /crucible/0/regions/c945c2c5-9fb1-49df-a99c-c51dce8b68af/snapshots/7f7566db-9859-4847-b89f-c07e9599a3b1/run