Max level shown:
12024-09-25T17:31:31.209ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:60181
22024-09-25T17:31:31.209ZINFOcrucible-agent (datafile): region 3d55d30b-706f-4976-bc9e-0a0102afe36a state: Requested
32024-09-25T17:31:31.210ZINFOcrucible-agent (dropshot): request completed latency_us = 868 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60181 req_id = c4f56210-aebd-42ad-8bee-e9ea1029e065 response_code = 200 uri = /crucible/0/regions
42024-09-25T17:31:31.210ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
52024-09-25T17:31:31.221ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_6cd0db00-82cc-45a5-9786-a174954df7f0/crucible/regions/3d55d30b-706f-4976-bc9e-0a0102afe36a
62024-09-25T17:31:31.221ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_6cd0db00-82cc-45a5-9786-a174954df7f0/crucible/regions/3d55d30b-706f-4976-bc9e-0a0102afe36a
72024-09-25T17:31:31.288ZINFOcrucible-agent (worker): creating region Region { id: RegionId("3d55d30b-706f-4976-bc9e-0a0102afe36a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/3d55d30b-706f-4976-bc9e-0a0102afe36a" region = 3d55d30b-706f-4976-bc9e-0a0102afe36a
82024-09-25T17:31:31.290ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "3d55d30b-706f-4976-bc9e-0a0102afe36a" "--data" "/data/regions/3d55d30b-706f-4976-bc9e-0a0102afe36a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 3d55d30b-706f-4976-bc9e-0a0102afe36a
92024-09-25T17:31:31.356ZINFOcrucible-agent (worker): region files created ok region = 3d55d30b-706f-4976-bc9e-0a0102afe36a
102024-09-25T17:31:31.356ZINFOcrucible-agent (datafile): region 3d55d30b-706f-4976-bc9e-0a0102afe36a state: Requested -> Created
112024-09-25T17:31:31.356ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-09-25T17:31:31.356ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-3d55d30b-706f-4976-bc9e-0a0102afe36a
132024-09-25T17:31:31.360ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-3d55d30b-706f-4976-bc9e-0a0102afe36a
142024-09-25T17:31:31.373ZINFOcrucible-agent (worker): creating config property group
152024-09-25T17:31:31.377ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3d55d30b-706f-4976-bc9e-0a0102afe36a
162024-09-25T17:31:31.377ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3d55d30b-706f-4976-bc9e-0a0102afe36a
172024-09-25T17:31:31.377ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182024-09-25T17:31:31.377ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
192024-09-25T17:31:31.377ZINFOcrucible-agent (worker): commit
202024-09-25T17:31:31.381ZINFOcrucible-agent (worker): ok!
212024-09-25T17:31:31.392ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/675a2946-3007-4c9a-90d5-d6352638af72
222024-09-25T17:31:31.392ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
232024-09-25T17:31:31.392ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::15
242024-09-25T17:31:31.392ZINFOcrucible-agent (worker): reconfig required
252024-09-25T17:31:31.392ZINFOcrucible-agent (worker): using existing config property group
262024-09-25T17:31:31.392ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-675a2946-3007-4c9a-90d5-d6352638af72
272024-09-25T17:31:31.392ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/675a2946-3007-4c9a-90d5-d6352638af72
282024-09-25T17:31:31.395ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-09-25T17:31:31.395ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
302024-09-25T17:31:31.395ZINFOcrucible-agent (worker): commit
312024-09-25T17:31:31.400ZINFOcrucible-agent (worker): ok!
322024-09-25T17:31:31.400ZINFOcrucible-agent (worker): SMF ok!
332024-09-25T17:31:31.545ZINFOcrucible-agent (dropshot): request completed latency_us = 808 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60181 req_id = 1ac66021-d8a5-4253-b2be-5a10f75f510c response_code = 200 uri = /crucible/0/regions
342024-09-25T17:31:33.130ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:60181 req_id = fc97f319-404c-4e33-ac6f-2381cb8c01e5 response_code = 200 uri = /crucible/0/regions/675a2946-3007-4c9a-90d5-d6352638af72
352024-09-25T17:31:33.134ZINFOcrucible-agent (datafile): path is "/data/regions/675a2946-3007-4c9a-90d5-d6352638af72"
362024-09-25T17:31:33.143ZINFOcrucible-agent (datafile): dataset is oxp_6cd0db00-82cc-45a5-9786-a174954df7f0/crucible/regions/675a2946-3007-4c9a-90d5-d6352638af72
372024-09-25T17:31:33.151ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_6cd0db00-82cc-45a5-9786-a174954df7f0/crucible/regions/675a2946-3007-4c9a-90d5-d6352638af72@63c69f1a-b579-4f6f-aa85-a4d0208cd7ff
382024-09-25T17:31:33.151ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_6cd0db00-82cc-45a5-9786-a174954df7f0/crucible/regions/675a2946-3007-4c9a-90d5-d6352638af72", "63c69f1a-b579-4f6f-aa85-a4d0208cd7ff"]
392024-09-25T17:31:33.161ZINFOcrucible-agent (dropshot): request completed latency_us = 27529 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:60181 req_id = 717878f0-95e8-4776-a7fa-d3668300b6c8 response_code = 200 uri = /crucible/0/regions/675a2946-3007-4c9a-90d5-d6352638af72/snapshots/63c69f1a-b579-4f6f-aa85-a4d0208cd7ff
402024-09-25T17:31:33.165ZINFOcrucible-agent (datafile): path is "/data/regions/675a2946-3007-4c9a-90d5-d6352638af72"
412024-09-25T17:31:33.173ZINFOcrucible-agent (datafile): dataset is oxp_6cd0db00-82cc-45a5-9786-a174954df7f0/crucible/regions/675a2946-3007-4c9a-90d5-d6352638af72
422024-09-25T17:31:33.180ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_6cd0db00-82cc-45a5-9786-a174954df7f0/crucible/regions/675a2946-3007-4c9a-90d5-d6352638af72@63c69f1a-b579-4f6f-aa85-a4d0208cd7ff
432024-09-25T17:31:33.180ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_6cd0db00-82cc-45a5-9786-a174954df7f0/crucible/regions/675a2946-3007-4c9a-90d5-d6352638af72", "63c69f1a-b579-4f6f-aa85-a4d0208cd7ff"]
442024-09-25T17:31:33.189ZINFOcrucible-agent (datafile): requesting running snapshot 675a2946-3007-4c9a-90d5-d6352638af72-63c69f1a-b579-4f6f-aa85-a4d0208cd7ff state: Requested
452024-09-25T17:31:33.190ZINFOcrucible-agent (worker): applying SMF actions for region 675a2946-3007-4c9a-90d5-d6352638af72 running snapshot 63c69f1a-b579-4f6f-aa85-a4d0208cd7ff (state Requested)...
462024-09-25T17:31:33.190ZINFOcrucible-agent (dropshot): request completed latency_us = 25067 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60181 req_id = 11e9526b-804d-4810-8448-44c3ec6226c6 response_code = 200 uri = /crucible/0/regions/675a2946-3007-4c9a-90d5-d6352638af72/snapshots/63c69f1a-b579-4f6f-aa85-a4d0208cd7ff/run
472024-09-25T17:31:33.191ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/675a2946-3007-4c9a-90d5-d6352638af72
482024-09-25T17:31:33.191ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-09-25T17:31:33.192ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::15
502024-09-25T17:31:33.192ZINFOcrucible-agent (worker): reconfig required
512024-09-25T17:31:33.192ZINFOcrucible-agent (worker): using existing config property group
522024-09-25T17:31:33.192ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-675a2946-3007-4c9a-90d5-d6352638af72
532024-09-25T17:31:33.192ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/675a2946-3007-4c9a-90d5-d6352638af72
542024-09-25T17:31:33.192ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-09-25T17:31:33.192ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
562024-09-25T17:31:33.192ZINFOcrucible-agent (worker): commit
572024-09-25T17:31:33.196ZINFOcrucible-agent (worker): ok!
582024-09-25T17:31:33.196ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-675a2946-3007-4c9a-90d5-d6352638af72-63c69f1a-b579-4f6f-aa85-a4d0208cd7ff
592024-09-25T17:31:33.202ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-675a2946-3007-4c9a-90d5-d6352638af72-63c69f1a-b579-4f6f-aa85-a4d0208cd7ff
602024-09-25T17:31:33.211ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-675a2946-3007-4c9a-90d5-d6352638af72-63c69f1a-b579-4f6f-aa85-a4d0208cd7ff
612024-09-25T17:31:33.212ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/675a2946-3007-4c9a-90d5-d6352638af72/.zfs/snapshot/63c69f1a-b579-4f6f-aa85-a4d0208cd7ff
622024-09-25T17:31:33.212ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
632024-09-25T17:31:33.212ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-09-25T17:31:33.212ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
652024-09-25T17:31:33.212ZINFOcrucible-agent (worker): commit
662024-09-25T17:31:33.216ZINFOcrucible-agent (worker): ok!
672024-09-25T17:31:33.220ZINFOcrucible-agent (worker): SMF ok!
682024-09-25T17:31:33.220ZINFOcrucible-agent (datafile): region 675a2946-3007-4c9a-90d5-d6352638af72 running snapshot 63c69f1a-b579-4f6f-aa85-a4d0208cd7ff state: Requested -> Created
692024-09-25T17:31:33.443ZINFOcrucible-agent (datafile): path is "/data/regions/675a2946-3007-4c9a-90d5-d6352638af72"
702024-09-25T17:31:33.452ZINFOcrucible-agent (datafile): dataset is oxp_6cd0db00-82cc-45a5-9786-a174954df7f0/crucible/regions/675a2946-3007-4c9a-90d5-d6352638af72
712024-09-25T17:31:33.461ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_6cd0db00-82cc-45a5-9786-a174954df7f0/crucible/regions/675a2946-3007-4c9a-90d5-d6352638af72@63c69f1a-b579-4f6f-aa85-a4d0208cd7ff
722024-09-25T17:31:33.461ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_6cd0db00-82cc-45a5-9786-a174954df7f0/crucible/regions/675a2946-3007-4c9a-90d5-d6352638af72", "63c69f1a-b579-4f6f-aa85-a4d0208cd7ff"]
732024-09-25T17:31:33.468ZINFOcrucible-agent (dropshot): request completed latency_us = 25435 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60181 req_id = f816c685-b6d0-45d4-ab3a-b0b60bbfa220 response_code = 200 uri = /crucible/0/regions/675a2946-3007-4c9a-90d5-d6352638af72/snapshots/63c69f1a-b579-4f6f-aa85-a4d0208cd7ff/run