Max level shown:
12024-05-30T04:16:21.152ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:46384
22024-05-30T04:16:21.152ZINFOcrucible-agent (datafile): region ecb72ad0-1906-4152-94e9-9e72a25f467a state: Requested
32024-05-30T04:16:21.153ZINFOcrucible-agent (dropshot): request completed latency_us = 766 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46384 req_id = 291259b8-f33c-412d-bfa9-54ea435c899b response_code = 200 uri = /crucible/0/regions
42024-05-30T04:16:21.153ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
52024-05-30T04:16:21.162ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_2efcf3a6-ef7f-48c2-8713-0dd19fe385d1/crucible/regions/ecb72ad0-1906-4152-94e9-9e72a25f467a
62024-05-30T04:16:21.162ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_2efcf3a6-ef7f-48c2-8713-0dd19fe385d1/crucible/regions/ecb72ad0-1906-4152-94e9-9e72a25f467a
72024-05-30T04:16:21.234ZINFOcrucible-agent (worker): creating region Region { id: RegionId("ecb72ad0-1906-4152-94e9-9e72a25f467a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/ecb72ad0-1906-4152-94e9-9e72a25f467a" region = ecb72ad0-1906-4152-94e9-9e72a25f467a
82024-05-30T04:16:21.236ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "ecb72ad0-1906-4152-94e9-9e72a25f467a" "--data" "/data/regions/ecb72ad0-1906-4152-94e9-9e72a25f467a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = ecb72ad0-1906-4152-94e9-9e72a25f467a
92024-05-30T04:16:21.301ZINFOcrucible-agent (worker): region files created ok region = ecb72ad0-1906-4152-94e9-9e72a25f467a
102024-05-30T04:16:21.301ZINFOcrucible-agent (datafile): region ecb72ad0-1906-4152-94e9-9e72a25f467a state: Requested -> Created
112024-05-30T04:16:21.301ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-05-30T04:16:21.301ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-ecb72ad0-1906-4152-94e9-9e72a25f467a
132024-05-30T04:16:21.306ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-ecb72ad0-1906-4152-94e9-9e72a25f467a
142024-05-30T04:16:21.320ZINFOcrucible-agent (worker): creating config property group
152024-05-30T04:16:21.323ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ecb72ad0-1906-4152-94e9-9e72a25f467a
162024-05-30T04:16:21.323ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ecb72ad0-1906-4152-94e9-9e72a25f467a
172024-05-30T04:16:21.323ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182024-05-30T04:16:21.323ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
192024-05-30T04:16:21.323ZINFOcrucible-agent (worker): commit
202024-05-30T04:16:21.326ZINFOcrucible-agent (worker): ok!
212024-05-30T04:16:21.331ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb
222024-05-30T04:16:21.331ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
232024-05-30T04:16:21.332ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
242024-05-30T04:16:21.332ZINFOcrucible-agent (worker): reconfig required
252024-05-30T04:16:21.332ZINFOcrucible-agent (worker): using existing config property group
262024-05-30T04:16:21.332ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ed87fbdc-c16c-4c5d-90cc-784636ca02cb
272024-05-30T04:16:21.332ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb
282024-05-30T04:16:21.332ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-05-30T04:16:21.332ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
302024-05-30T04:16:21.332ZINFOcrucible-agent (worker): commit
312024-05-30T04:16:21.341ZINFOcrucible-agent (worker): ok!
322024-05-30T04:16:21.341ZINFOcrucible-agent (worker): SMF ok!
332024-05-30T04:16:21.348ZINFOcrucible-agent (dropshot): request completed latency_us = 682 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46384 req_id = ba6d22a0-1ffd-491c-aae1-bdc2bda000e2 response_code = 200 uri = /crucible/0/regions
342024-05-30T04:16:22.708ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:44464
352024-05-30T04:16:22.710ZINFOcrucible-agent (dropshot): request completed latency_us = 2888 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:44464 req_id = 644f09c9-ed15-4d6b-9d96-6c7b01bdf0a8 response_code = 200 uri = /crucible/0/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb
362024-05-30T04:16:22.711ZINFOcrucible-agent (datafile): path is "/data/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb"
372024-05-30T04:16:22.720ZINFOcrucible-agent (datafile): dataset is oxp_2efcf3a6-ef7f-48c2-8713-0dd19fe385d1/crucible/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb
382024-05-30T04:16:22.730ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_2efcf3a6-ef7f-48c2-8713-0dd19fe385d1/crucible/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb@96b1f498-e4fa-43b0-89db-290a75012292
392024-05-30T04:16:22.730ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_2efcf3a6-ef7f-48c2-8713-0dd19fe385d1/crucible/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb", "96b1f498-e4fa-43b0-89db-290a75012292"]
402024-05-30T04:16:22.743ZINFOcrucible-agent (dropshot): request completed latency_us = 32564 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:44464 req_id = f59a0133-783c-446f-ae02-3b1b511a06ee response_code = 200 uri = /crucible/0/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb/snapshots/96b1f498-e4fa-43b0-89db-290a75012292
412024-05-30T04:16:22.743ZINFOcrucible-agent (datafile): path is "/data/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb"
422024-05-30T04:16:22.752ZINFOcrucible-agent (datafile): dataset is oxp_2efcf3a6-ef7f-48c2-8713-0dd19fe385d1/crucible/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb
432024-05-30T04:16:22.760ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_2efcf3a6-ef7f-48c2-8713-0dd19fe385d1/crucible/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb@96b1f498-e4fa-43b0-89db-290a75012292
442024-05-30T04:16:22.760ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_2efcf3a6-ef7f-48c2-8713-0dd19fe385d1/crucible/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb", "96b1f498-e4fa-43b0-89db-290a75012292"]
452024-05-30T04:16:22.769ZINFOcrucible-agent (datafile): requesting running snapshot ed87fbdc-c16c-4c5d-90cc-784636ca02cb-96b1f498-e4fa-43b0-89db-290a75012292 state: Requested
462024-05-30T04:16:22.769ZINFOcrucible-agent (worker): applying SMF actions for region ed87fbdc-c16c-4c5d-90cc-784636ca02cb running snapshot 96b1f498-e4fa-43b0-89db-290a75012292 (state Requested)...
472024-05-30T04:16:22.769ZINFOcrucible-agent (dropshot): request completed latency_us = 25976 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:44464 req_id = c591f18e-90b9-4303-a4ab-ed0e5734dccf response_code = 200 uri = /crucible/0/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb/snapshots/96b1f498-e4fa-43b0-89db-290a75012292/run
482024-05-30T04:16:22.771ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb
492024-05-30T04:16:22.771ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
502024-05-30T04:16:22.771ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
512024-05-30T04:16:22.771ZINFOcrucible-agent (worker): reconfig required
522024-05-30T04:16:22.771ZINFOcrucible-agent (worker): using existing config property group
532024-05-30T04:16:22.771ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ed87fbdc-c16c-4c5d-90cc-784636ca02cb
542024-05-30T04:16:22.772ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb
552024-05-30T04:16:22.772ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
562024-05-30T04:16:22.772ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
572024-05-30T04:16:22.772ZINFOcrucible-agent (worker): commit
582024-05-30T04:16:22.775ZINFOcrucible-agent (worker): ok!
592024-05-30T04:16:22.775ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-ed87fbdc-c16c-4c5d-90cc-784636ca02cb-96b1f498-e4fa-43b0-89db-290a75012292
602024-05-30T04:16:22.781ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-ed87fbdc-c16c-4c5d-90cc-784636ca02cb-96b1f498-e4fa-43b0-89db-290a75012292
612024-05-30T04:16:22.790ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-ed87fbdc-c16c-4c5d-90cc-784636ca02cb-96b1f498-e4fa-43b0-89db-290a75012292
622024-05-30T04:16:22.791ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb/.zfs/snapshot/96b1f498-e4fa-43b0-89db-290a75012292
632024-05-30T04:16:22.791ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
642024-05-30T04:16:22.791ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
652024-05-30T04:16:22.791ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
662024-05-30T04:16:22.791ZINFOcrucible-agent (worker): commit
672024-05-30T04:16:22.794ZINFOcrucible-agent (worker): ok!
682024-05-30T04:16:22.798ZINFOcrucible-agent (worker): SMF ok!
692024-05-30T04:16:22.798ZINFOcrucible-agent (datafile): region ed87fbdc-c16c-4c5d-90cc-784636ca02cb running snapshot 96b1f498-e4fa-43b0-89db-290a75012292 state: Requested -> Created
702024-05-30T04:16:25.324ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:32791
712024-05-30T04:16:25.324ZINFOcrucible-agent (datafile): region b0312434-8481-417d-be57-763a8f4502ab state: Requested
722024-05-30T04:16:25.324ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
732024-05-30T04:16:25.324ZINFOcrucible-agent (dropshot): request completed latency_us = 334 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:32791 req_id = 6d04816f-2741-45a5-a649-9fa23b75d96c response_code = 200 uri = /crucible/0/regions
742024-05-30T04:16:25.334ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_2efcf3a6-ef7f-48c2-8713-0dd19fe385d1/crucible/regions/b0312434-8481-417d-be57-763a8f4502ab
752024-05-30T04:16:25.334ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_2efcf3a6-ef7f-48c2-8713-0dd19fe385d1/crucible/regions/b0312434-8481-417d-be57-763a8f4502ab
762024-05-30T04:16:25.407ZINFOcrucible-agent (worker): creating region Region { id: RegionId("b0312434-8481-417d-be57-763a8f4502ab"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/b0312434-8481-417d-be57-763a8f4502ab" region = b0312434-8481-417d-be57-763a8f4502ab
772024-05-30T04:16:25.410ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "b0312434-8481-417d-be57-763a8f4502ab" "--data" "/data/regions/b0312434-8481-417d-be57-763a8f4502ab" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = b0312434-8481-417d-be57-763a8f4502ab
782024-05-30T04:16:25.476ZINFOcrucible-agent (worker): region files created ok region = b0312434-8481-417d-be57-763a8f4502ab
792024-05-30T04:16:25.476ZINFOcrucible-agent (datafile): region b0312434-8481-417d-be57-763a8f4502ab state: Requested -> Created
802024-05-30T04:16:25.477ZINFOcrucible-agent (worker): applying SMF actions post create...
812024-05-30T04:16:25.477ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-b0312434-8481-417d-be57-763a8f4502ab
822024-05-30T04:16:25.480ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-b0312434-8481-417d-be57-763a8f4502ab
832024-05-30T04:16:25.487ZINFOcrucible-agent (worker): creating config property group
842024-05-30T04:16:25.491ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-b0312434-8481-417d-be57-763a8f4502ab
852024-05-30T04:16:25.491ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b0312434-8481-417d-be57-763a8f4502ab
862024-05-30T04:16:25.491ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
872024-05-30T04:16:25.491ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
882024-05-30T04:16:25.491ZINFOcrucible-agent (worker): commit
892024-05-30T04:16:25.495ZINFOcrucible-agent (worker): ok!
902024-05-30T04:16:25.500ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb
912024-05-30T04:16:25.500ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
922024-05-30T04:16:25.500ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
932024-05-30T04:16:25.500ZINFOcrucible-agent (worker): reconfig required
942024-05-30T04:16:25.500ZINFOcrucible-agent (worker): using existing config property group
952024-05-30T04:16:25.500ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ed87fbdc-c16c-4c5d-90cc-784636ca02cb
962024-05-30T04:16:25.504ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb
972024-05-30T04:16:25.504ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
982024-05-30T04:16:25.504ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
992024-05-30T04:16:25.504ZINFOcrucible-agent (worker): commit
1002024-05-30T04:16:25.508ZINFOcrucible-agent (worker): ok!
1012024-05-30T04:16:25.510ZINFOcrucible-agent (worker): SMF ok!
1022024-05-30T04:16:25.513ZINFOcrucible-agent (dropshot): request completed latency_us = 75 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:32791 req_id = 973e20b3-9e49-41dd-bdcd-9f83f18f1457 response_code = 200 uri = /crucible/0/regions
1032024-05-30T04:16:48.775ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:58509
1042024-05-30T04:16:48.775ZINFOcrucible-agent (dropshot): request completed latency_us = 47 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:58509 req_id = 098dac52-e244-4fe7-bf6e-087a7252d70f response_code = 200 uri = /crucible/0/regions/b0312434-8481-417d-be57-763a8f4502ab
1052024-05-30T04:16:48.775ZINFOcrucible-agent (datafile): path is "/data/regions/b0312434-8481-417d-be57-763a8f4502ab"
1062024-05-30T04:16:48.786ZINFOcrucible-agent (datafile): dataset is oxp_2efcf3a6-ef7f-48c2-8713-0dd19fe385d1/crucible/regions/b0312434-8481-417d-be57-763a8f4502ab
1072024-05-30T04:16:48.793ZINFOcrucible-agent (datafile): region b0312434-8481-417d-be57-763a8f4502ab state: Created -> Tombstoned
1082024-05-30T04:16:48.794ZINFOcrucible-agent (worker): applying SMF actions before removal...
1092024-05-30T04:16:48.796ZINFOcrucible-agent (dropshot): request completed latency_us = 21214 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:58509 req_id = 1c06b176-4f47-4a91-9b9b-ec1aa59eeb43 response_code = 204 uri = /crucible/0/regions/b0312434-8481-417d-be57-763a8f4502ab
1102024-05-30T04:16:48.796ZINFOcrucible-agent (dropshot): request completed latency_us = 25 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:58509 req_id = 36eea0c2-ddef-43c9-91e6-db8233338ab9 response_code = 200 uri = /crucible/0/regions/b0312434-8481-417d-be57-763a8f4502ab
1112024-05-30T04:16:48.797ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-b0312434-8481-417d-be57-763a8f4502ab (instance states: (Some(Online), None))
1122024-05-30T04:16:48.805ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb
1132024-05-30T04:16:48.805ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1142024-05-30T04:16:48.805ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
1152024-05-30T04:16:48.805ZINFOcrucible-agent (worker): reconfig required
1162024-05-30T04:16:48.805ZINFOcrucible-agent (worker): using existing config property group
1172024-05-30T04:16:48.805ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ed87fbdc-c16c-4c5d-90cc-784636ca02cb
1182024-05-30T04:16:48.805ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ed87fbdc-c16c-4c5d-90cc-784636ca02cb
1192024-05-30T04:16:48.805ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1202024-05-30T04:16:48.805ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
1212024-05-30T04:16:48.805ZINFOcrucible-agent (worker): commit
1222024-05-30T04:16:48.809ZINFOcrucible-agent (worker): ok!
1232024-05-30T04:16:48.810ZINFOcrucible-agent (worker): SMF ok!
1242024-05-30T04:16:48.820ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_2efcf3a6-ef7f-48c2-8713-0dd19fe385d1/crucible/regions/b0312434-8481-417d-be57-763a8f4502ab" region = b0312434-8481-417d-be57-763a8f4502ab
1252024-05-30T04:16:48.874ZINFOcrucible-agent (datafile): region b0312434-8481-417d-be57-763a8f4502ab state: Tombstoned -> Destroyed
1262024-05-30T04:16:48.946ZINFOcrucible-agent (dropshot): request completed latency_us = 943 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:58509 req_id = c7531255-9a90-4fec-94e9-a640ee88f333 response_code = 200 uri = /crucible/0/regions/b0312434-8481-417d-be57-763a8f4502ab