Max level shown:
12024-02-12T21:16:19.725ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:42632
22024-02-12T21:16:19.805ZINFOcrucible-agent (datafile): region 2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9 state: Requested
32024-02-12T21:16:19.808ZINFOcrucible-agent (dropshot): request completed latency_us = 13088 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42632 req_id = bd7772a3-9de7-4528-b410-9c7b4039acf9 response_code = 200 uri = /crucible/0/regions
42024-02-12T21:16:19.812ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
52024-02-12T21:16:19.822ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
62024-02-12T21:16:19.822ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
72024-02-12T21:16:19.868ZINFOcrucible-agent (worker): creating region Region { id: RegionId("2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9" region = 2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
82024-02-12T21:16:20.165ZINFOcrucible-agent (worker): region files created ok region = 2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
92024-02-12T21:16:20.165ZINFOcrucible-agent (datafile): region 2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9 state: Requested -> Created
102024-02-12T21:16:20.166ZINFOcrucible-agent (worker): applying SMF actions post create...
112024-02-12T21:16:20.166ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
122024-02-12T21:16:20.169ZINFOcrucible-agent (dropshot): request completed latency_us = 79 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42632 req_id = 4e5a4d5b-f8a2-47a5-9e42-bb5f7e25ae06 response_code = 200 uri = /crucible/0/regions
132024-02-12T21:16:20.173ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
142024-02-12T21:16:20.188ZINFOcrucible-agent (worker): creating config property group
152024-02-12T21:16:20.194ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
162024-02-12T21:16:20.195ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
172024-02-12T21:16:20.197ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
182024-02-12T21:16:20.197ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
192024-02-12T21:16:20.197ZINFOcrucible-agent (worker): commit
202024-02-12T21:16:20.211ZINFOcrucible-agent (worker): ok!
212024-02-12T21:16:20.216ZINFOcrucible-agent (worker): SMF ok!
222024-02-12T21:21:49.003ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:48727
232024-02-12T21:21:49.004ZINFOcrucible-agent (datafile): region ca3ceb9a-0fac-4024-86be-68eec8e8446a state: Requested
242024-02-12T21:21:49.008ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
252024-02-12T21:21:49.008ZINFOcrucible-agent (dropshot): request completed latency_us = 4037 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:48727 req_id = 6c3e200b-a7cf-4e29-bc89-25bbf826b886 response_code = 200 uri = /crucible/0/regions
262024-02-12T21:21:49.020ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/ca3ceb9a-0fac-4024-86be-68eec8e8446a
272024-02-12T21:21:49.020ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/ca3ceb9a-0fac-4024-86be-68eec8e8446a
282024-02-12T21:21:49.081ZINFOcrucible-agent (worker): creating region Region { id: RegionId("ca3ceb9a-0fac-4024-86be-68eec8e8446a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/ca3ceb9a-0fac-4024-86be-68eec8e8446a" region = ca3ceb9a-0fac-4024-86be-68eec8e8446a
292024-02-12T21:21:49.202ZINFOcrucible-agent (worker): region files created ok region = ca3ceb9a-0fac-4024-86be-68eec8e8446a
302024-02-12T21:21:49.202ZINFOcrucible-agent (datafile): region ca3ceb9a-0fac-4024-86be-68eec8e8446a state: Requested -> Created
312024-02-12T21:21:49.202ZINFOcrucible-agent (worker): applying SMF actions post create...
322024-02-12T21:21:49.208ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
332024-02-12T21:21:49.209ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
342024-02-12T21:21:49.209ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
352024-02-12T21:21:49.209ZINFOcrucible-agent (worker): reconfig required
362024-02-12T21:21:49.209ZINFOcrucible-agent (worker): using existing config property group
372024-02-12T21:21:49.209ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
382024-02-12T21:21:49.209ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
392024-02-12T21:21:49.209ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
402024-02-12T21:21:49.209ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
412024-02-12T21:21:49.209ZINFOcrucible-agent (worker): commit
422024-02-12T21:21:49.221ZINFOcrucible-agent (worker): ok!
432024-02-12T21:21:49.222ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-ca3ceb9a-0fac-4024-86be-68eec8e8446a
442024-02-12T21:21:49.229ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-ca3ceb9a-0fac-4024-86be-68eec8e8446a
452024-02-12T21:21:49.241ZINFOcrucible-agent (worker): creating config property group
462024-02-12T21:21:49.247ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ca3ceb9a-0fac-4024-86be-68eec8e8446a
472024-02-12T21:21:49.248ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ca3ceb9a-0fac-4024-86be-68eec8e8446a
482024-02-12T21:21:49.248ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
492024-02-12T21:21:49.248ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
502024-02-12T21:21:49.248ZINFOcrucible-agent (worker): commit
512024-02-12T21:21:49.253ZINFOcrucible-agent (worker): ok!
522024-02-12T21:21:49.260ZINFOcrucible-agent (worker): SMF ok!
532024-02-12T21:21:49.316ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:48727 req_id = 710595b1-3d2f-4db5-902b-f77ff6856f3b response_code = 200 uri = /crucible/0/regions
542024-02-12T21:21:50.467ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:48486
552024-02-12T21:21:50.468ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:48486 req_id = 5d349997-bf24-430a-a2bd-86f88af454af response_code = 200 uri = /crucible/0/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
562024-02-12T21:21:50.470ZINFOcrucible-agent (datafile): path is "/data/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9"
572024-02-12T21:21:50.480ZINFOcrucible-agent (datafile): dataset is oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
582024-02-12T21:21:50.488ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9@d5595bbd-d9b8-4f25-96e0-b133935ed4b1
592024-02-12T21:21:50.488ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9", "d5595bbd-d9b8-4f25-96e0-b133935ed4b1"]
602024-02-12T21:21:50.499ZINFOcrucible-agent (dropshot): request completed latency_us = 31139 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:48486 req_id = d3ab0e59-b9d0-4e0e-8869-27ee491793af response_code = 200 uri = /crucible/0/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9/snapshots/d5595bbd-d9b8-4f25-96e0-b133935ed4b1
612024-02-12T21:21:50.501ZINFOcrucible-agent (datafile): path is "/data/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9"
622024-02-12T21:21:50.509ZINFOcrucible-agent (datafile): dataset is oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
632024-02-12T21:21:50.517ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9@d5595bbd-d9b8-4f25-96e0-b133935ed4b1
642024-02-12T21:21:50.517ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9", "d5595bbd-d9b8-4f25-96e0-b133935ed4b1"]
652024-02-12T21:21:50.524ZINFOcrucible-agent (datafile): requesting running snapshot 2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9-d5595bbd-d9b8-4f25-96e0-b133935ed4b1 state: Requested
662024-02-12T21:21:50.524ZINFOcrucible-agent (worker): applying SMF actions for region 2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9 running snapshot d5595bbd-d9b8-4f25-96e0-b133935ed4b1 (state Requested)...
672024-02-12T21:21:50.524ZINFOcrucible-agent (dropshot): request completed latency_us = 23090 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:48486 req_id = 1671681e-f674-49ad-ba94-904e071d830a response_code = 200 uri = /crucible/0/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9/snapshots/d5595bbd-d9b8-4f25-96e0-b133935ed4b1/run
682024-02-12T21:21:50.525ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
692024-02-12T21:21:50.525ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
702024-02-12T21:21:50.525ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
712024-02-12T21:21:50.525ZINFOcrucible-agent (worker): reconfig required
722024-02-12T21:21:50.525ZINFOcrucible-agent (worker): using existing config property group
732024-02-12T21:21:50.525ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
742024-02-12T21:21:50.526ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9
752024-02-12T21:21:50.526ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
762024-02-12T21:21:50.526ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
772024-02-12T21:21:50.526ZINFOcrucible-agent (worker): commit
782024-02-12T21:21:50.532ZINFOcrucible-agent (worker): ok!
792024-02-12T21:21:50.534ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9-d5595bbd-d9b8-4f25-96e0-b133935ed4b1
802024-02-12T21:21:50.543ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9-d5595bbd-d9b8-4f25-96e0-b133935ed4b1
812024-02-12T21:21:50.561ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9-d5595bbd-d9b8-4f25-96e0-b133935ed4b1
822024-02-12T21:21:50.561ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9/.zfs/snapshot/d5595bbd-d9b8-4f25-96e0-b133935ed4b1
832024-02-12T21:21:50.561ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
842024-02-12T21:21:50.561ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
852024-02-12T21:21:50.561ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
862024-02-12T21:21:50.561ZINFOcrucible-agent (worker): commit
872024-02-12T21:21:50.568ZINFOcrucible-agent (worker): ok!
882024-02-12T21:21:50.573ZINFOcrucible-agent (worker): SMF ok!
892024-02-12T21:21:50.573ZINFOcrucible-agent (datafile): region 2cc91bce-8d47-4bda-b2fe-b6fd1fed54f9 running snapshot d5595bbd-d9b8-4f25-96e0-b133935ed4b1 state: Requested -> Created