Max level shown:
1 [ Jun 17 18:06:38 Enabled. ]
2 [ Jun 17 18:06:38 Rereading configuration. ]
3 [ Jun 17 18:06:39 Rereading configuration. ]
4 [ Jun 17 18:06:41 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_3fece30f-d49f-45ba-91b6-69d5deecf97a/crucible -l [fd00:1122:3344:101::1a]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-06-17T18:06:41.491ZINFOcrucible-agent: dataset: "oxp_3fece30f-d49f-45ba-91b6-69d5deecf97a/crucible"
72024-06-17T18:06:41.494ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::1a]:32345
82024-06-17T18:06:41.494ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-06-17T18:06:41.515ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-06-17T18:06:41.599ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::1a]:32345
112024-06-17T18:12:23.861ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::a]:57541
122024-06-17T18:12:23.898ZINFOcrucible-agent (datafile): region 838292a5-e31c-42cf-be81-6d10da31ec07 state: Requested
132024-06-17T18:12:23.906ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
142024-06-17T18:12:23.909ZINFOcrucible-agent (dropshot): request completed latency_us = 21103 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57541 req_id = 43869e45-d462-4d9d-aa77-92197b987662 response_code = 200 uri = /crucible/0/regions
152024-06-17T18:12:23.916ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_3fece30f-d49f-45ba-91b6-69d5deecf97a/crucible/regions/838292a5-e31c-42cf-be81-6d10da31ec07
162024-06-17T18:12:23.916ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_3fece30f-d49f-45ba-91b6-69d5deecf97a/crucible/regions/838292a5-e31c-42cf-be81-6d10da31ec07
172024-06-17T18:12:23.973ZINFOcrucible-agent (worker): creating region Region { id: RegionId("838292a5-e31c-42cf-be81-6d10da31ec07"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/838292a5-e31c-42cf-be81-6d10da31ec07" region = 838292a5-e31c-42cf-be81-6d10da31ec07
182024-06-17T18:12:23.975ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "838292a5-e31c-42cf-be81-6d10da31ec07" "--data" "/data/regions/838292a5-e31c-42cf-be81-6d10da31ec07" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 838292a5-e31c-42cf-be81-6d10da31ec07
192024-06-17T18:12:24.071ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57541 req_id = 2af406cc-6ecc-431c-9dfb-136dc4a2c4cd response_code = 200 uri = /crucible/0/regions
202024-06-17T18:12:24.296ZINFOcrucible-agent (worker): region files created ok region = 838292a5-e31c-42cf-be81-6d10da31ec07
212024-06-17T18:12:24.296ZINFOcrucible-agent (datafile): region 838292a5-e31c-42cf-be81-6d10da31ec07 state: Requested -> Created
222024-06-17T18:12:24.297ZINFOcrucible-agent (worker): applying SMF actions post create...
232024-06-17T18:12:24.297ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-838292a5-e31c-42cf-be81-6d10da31ec07
242024-06-17T18:12:24.304ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-838292a5-e31c-42cf-be81-6d10da31ec07
252024-06-17T18:12:24.316ZINFOcrucible-agent (worker): creating config property group
262024-06-17T18:12:24.321ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-838292a5-e31c-42cf-be81-6d10da31ec07
272024-06-17T18:12:24.328ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/838292a5-e31c-42cf-be81-6d10da31ec07
282024-06-17T18:12:24.330ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-06-17T18:12:24.330ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
302024-06-17T18:12:24.330ZINFOcrucible-agent (worker): commit
312024-06-17T18:12:24.335ZINFOcrucible-agent (worker): ok!
322024-06-17T18:12:24.338ZINFOcrucible-agent (worker): SMF ok!
332024-06-17T18:12:24.620ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57541 req_id = 94934328-1c62-4063-879a-e582221c216b response_code = 200 uri = /crucible/0/regions
342024-06-17T18:12:27.728ZINFOcrucible-agent (datafile): region ca828974-30f7-4249-9210-57ec774ae1d6 state: Requested
352024-06-17T18:12:27.729ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
362024-06-17T18:12:27.729ZINFOcrucible-agent (dropshot): request completed latency_us = 192 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57541 req_id = da874c00-9ab9-4f9e-9694-37ae065b7b80 response_code = 200 uri = /crucible/0/regions
372024-06-17T18:12:27.738ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_3fece30f-d49f-45ba-91b6-69d5deecf97a/crucible/regions/ca828974-30f7-4249-9210-57ec774ae1d6
382024-06-17T18:12:27.738ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_3fece30f-d49f-45ba-91b6-69d5deecf97a/crucible/regions/ca828974-30f7-4249-9210-57ec774ae1d6
392024-06-17T18:12:27.792ZINFOcrucible-agent (worker): creating region Region { id: RegionId("ca828974-30f7-4249-9210-57ec774ae1d6"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/ca828974-30f7-4249-9210-57ec774ae1d6" region = ca828974-30f7-4249-9210-57ec774ae1d6
402024-06-17T18:12:27.792ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "ca828974-30f7-4249-9210-57ec774ae1d6" "--data" "/data/regions/ca828974-30f7-4249-9210-57ec774ae1d6" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = ca828974-30f7-4249-9210-57ec774ae1d6
412024-06-17T18:12:27.863ZINFOcrucible-agent (worker): region files created ok region = ca828974-30f7-4249-9210-57ec774ae1d6
422024-06-17T18:12:27.863ZINFOcrucible-agent (datafile): region ca828974-30f7-4249-9210-57ec774ae1d6 state: Requested -> Created
432024-06-17T18:12:27.863ZINFOcrucible-agent (worker): applying SMF actions post create...
442024-06-17T18:12:27.864ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/838292a5-e31c-42cf-be81-6d10da31ec07
452024-06-17T18:12:27.864ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
462024-06-17T18:12:27.864ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::1a
472024-06-17T18:12:27.864ZINFOcrucible-agent (worker): reconfig required
482024-06-17T18:12:27.864ZINFOcrucible-agent (worker): using existing config property group
492024-06-17T18:12:27.864ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-838292a5-e31c-42cf-be81-6d10da31ec07
502024-06-17T18:12:27.864ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/838292a5-e31c-42cf-be81-6d10da31ec07
512024-06-17T18:12:27.864ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
522024-06-17T18:12:27.864ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
532024-06-17T18:12:27.864ZINFOcrucible-agent (worker): commit
542024-06-17T18:12:27.868ZINFOcrucible-agent (worker): ok!
552024-06-17T18:12:27.869ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-ca828974-30f7-4249-9210-57ec774ae1d6
562024-06-17T18:12:27.871ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-ca828974-30f7-4249-9210-57ec774ae1d6
572024-06-17T18:12:27.879ZINFOcrucible-agent (worker): creating config property group
582024-06-17T18:12:27.882ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ca828974-30f7-4249-9210-57ec774ae1d6
592024-06-17T18:12:27.882ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ca828974-30f7-4249-9210-57ec774ae1d6
602024-06-17T18:12:27.882ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
612024-06-17T18:12:27.882ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
622024-06-17T18:12:27.882ZINFOcrucible-agent (worker): commit
632024-06-17T18:12:27.885ZINFOcrucible-agent (worker): ok!
642024-06-17T18:12:27.890ZINFOcrucible-agent (worker): SMF ok!
652024-06-17T18:12:27.922ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57541 req_id = c208f8d0-5c62-4947-be98-59922f0792ef response_code = 200 uri = /crucible/0/regions
662024-06-17T18:12:51.759ZINFOcrucible-agent (dropshot): request completed latency_us = 106 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57541 req_id = 08dc063a-4552-4455-a33d-19a055afaf0d response_code = 200 uri = /crucible/0/regions/ca828974-30f7-4249-9210-57ec774ae1d6
672024-06-17T18:12:51.762ZINFOcrucible-agent (datafile): path is "/data/regions/ca828974-30f7-4249-9210-57ec774ae1d6"
682024-06-17T18:12:51.771ZINFOcrucible-agent (datafile): dataset is oxp_3fece30f-d49f-45ba-91b6-69d5deecf97a/crucible/regions/ca828974-30f7-4249-9210-57ec774ae1d6
692024-06-17T18:12:51.779ZINFOcrucible-agent (datafile): region ca828974-30f7-4249-9210-57ec774ae1d6 state: Created -> Tombstoned
702024-06-17T18:12:51.779ZINFOcrucible-agent (worker): applying SMF actions before removal...
712024-06-17T18:12:51.781ZINFOcrucible-agent (dropshot): request completed latency_us = 19652 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:57541 req_id = f3bd50bd-334f-40ff-8970-f8041f95c9f7 response_code = 204 uri = /crucible/0/regions/ca828974-30f7-4249-9210-57ec774ae1d6
722024-06-17T18:12:51.782ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-ca828974-30f7-4249-9210-57ec774ae1d6 (instance states: (Some(Online), None))
732024-06-17T18:12:51.784ZINFOcrucible-agent (dropshot): request completed latency_us = 23 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57541 req_id = f625d1b0-c23b-4d9e-b172-e72dfb20c264 response_code = 200 uri = /crucible/0/regions/ca828974-30f7-4249-9210-57ec774ae1d6
742024-06-17T18:12:51.787ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/838292a5-e31c-42cf-be81-6d10da31ec07
752024-06-17T18:12:51.787ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
762024-06-17T18:12:51.787ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::1a
772024-06-17T18:12:51.787ZINFOcrucible-agent (worker): reconfig required
782024-06-17T18:12:51.787ZINFOcrucible-agent (worker): using existing config property group
792024-06-17T18:12:51.787ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-838292a5-e31c-42cf-be81-6d10da31ec07
802024-06-17T18:12:51.787ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/838292a5-e31c-42cf-be81-6d10da31ec07
812024-06-17T18:12:51.787ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
822024-06-17T18:12:51.787ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
832024-06-17T18:12:51.787ZINFOcrucible-agent (worker): commit
842024-06-17T18:12:51.791ZINFOcrucible-agent (worker): ok!
852024-06-17T18:12:51.792ZINFOcrucible-agent (worker): SMF ok!
862024-06-17T18:12:51.801ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_3fece30f-d49f-45ba-91b6-69d5deecf97a/crucible/regions/ca828974-30f7-4249-9210-57ec774ae1d6" region = ca828974-30f7-4249-9210-57ec774ae1d6
872024-06-17T18:12:51.851ZINFOcrucible-agent (datafile): region ca828974-30f7-4249-9210-57ec774ae1d6 state: Tombstoned -> Destroyed
882024-06-17T18:12:51.910ZINFOcrucible-agent (dropshot): request completed latency_us = 742 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57541 req_id = 4cd4f9a1-e919-4cc7-9eca-1977caf31e75 response_code = 200 uri = /crucible/0/regions/ca828974-30f7-4249-9210-57ec774ae1d6