Max level shown:
12024-07-29T22:46:15.248ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:59948
22024-07-29T22:46:15.249ZINFOcrucible-agent (datafile): region 6249c043-ac12-4325-b6db-420386afbb19 state: Requested
32024-07-29T22:46:15.251ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-07-29T22:46:15.251ZINFOcrucible-agent (dropshot): request completed latency_us = 2895 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:59948 req_id = 9763486d-d2d7-4534-aac1-8f637a77af14 response_code = 200 uri = /crucible/0/regions
52024-07-29T22:46:15.262ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_04fcb670-76ba-4fb0-bafa-c9b2df44923c/crucible/regions/6249c043-ac12-4325-b6db-420386afbb19
62024-07-29T22:46:15.263ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_04fcb670-76ba-4fb0-bafa-c9b2df44923c/crucible/regions/6249c043-ac12-4325-b6db-420386afbb19
72024-07-29T22:46:15.316ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6249c043-ac12-4325-b6db-420386afbb19"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/6249c043-ac12-4325-b6db-420386afbb19" region = 6249c043-ac12-4325-b6db-420386afbb19
82024-07-29T22:46:15.316ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "6249c043-ac12-4325-b6db-420386afbb19" "--data" "/data/regions/6249c043-ac12-4325-b6db-420386afbb19" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 6249c043-ac12-4325-b6db-420386afbb19
92024-07-29T22:46:15.383ZINFOcrucible-agent (worker): region files created ok region = 6249c043-ac12-4325-b6db-420386afbb19
102024-07-29T22:46:15.383ZINFOcrucible-agent (datafile): region 6249c043-ac12-4325-b6db-420386afbb19 state: Requested -> Created
112024-07-29T22:46:15.383ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-07-29T22:46:15.384ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6249c043-ac12-4325-b6db-420386afbb19
132024-07-29T22:46:15.389ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6249c043-ac12-4325-b6db-420386afbb19
142024-07-29T22:46:15.402ZINFOcrucible-agent (worker): creating config property group
152024-07-29T22:46:15.406ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6249c043-ac12-4325-b6db-420386afbb19
162024-07-29T22:46:15.406ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6249c043-ac12-4325-b6db-420386afbb19
172024-07-29T22:46:15.406ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182024-07-29T22:46:15.406ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
192024-07-29T22:46:15.406ZINFOcrucible-agent (worker): commit
202024-07-29T22:46:15.406ZINFOcrucible-agent (dropshot): request completed latency_us = 68 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:59948 req_id = d7efbe9a-11fa-48ba-825f-b0ee7d33440c response_code = 200 uri = /crucible/0/regions
212024-07-29T22:46:15.411ZINFOcrucible-agent (worker): ok!
222024-07-29T22:46:15.418ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7
232024-07-29T22:46:15.419ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
242024-07-29T22:46:15.419ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::17
252024-07-29T22:46:15.419ZINFOcrucible-agent (worker): reconfig required
262024-07-29T22:46:15.419ZINFOcrucible-agent (worker): using existing config property group
272024-07-29T22:46:15.419ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7
282024-07-29T22:46:15.419ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7
292024-07-29T22:46:15.419ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302024-07-29T22:46:15.419ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
312024-07-29T22:46:15.419ZINFOcrucible-agent (worker): commit
322024-07-29T22:46:15.427ZINFOcrucible-agent (worker): ok!
332024-07-29T22:46:15.428ZINFOcrucible-agent (worker): SMF ok!
342024-07-29T22:46:16.500ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:36775
352024-07-29T22:46:16.503ZINFOcrucible-agent (dropshot): request completed latency_us = 2362 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36775 req_id = 6e7301dc-a09e-4786-a736-54eeea62cf7f response_code = 200 uri = /crucible/0/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7
362024-07-29T22:46:16.506ZINFOcrucible-agent (datafile): path is "/data/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7"
372024-07-29T22:46:16.515ZINFOcrucible-agent (datafile): dataset is oxp_04fcb670-76ba-4fb0-bafa-c9b2df44923c/crucible/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7
382024-07-29T22:46:16.525ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_04fcb670-76ba-4fb0-bafa-c9b2df44923c/crucible/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7@fdbce13f-1958-413b-b274-4950ad3f94e7
392024-07-29T22:46:16.525ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_04fcb670-76ba-4fb0-bafa-c9b2df44923c/crucible/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7", "fdbce13f-1958-413b-b274-4950ad3f94e7"]
402024-07-29T22:46:16.537ZINFOcrucible-agent (dropshot): request completed latency_us = 33657 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36775 req_id = 33cdd88b-b396-49ce-b1d6-e04efeccd335 response_code = 200 uri = /crucible/0/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7/snapshots/fdbce13f-1958-413b-b274-4950ad3f94e7
412024-07-29T22:46:16.537ZINFOcrucible-agent (datafile): path is "/data/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7"
422024-07-29T22:46:16.546ZINFOcrucible-agent (datafile): dataset is oxp_04fcb670-76ba-4fb0-bafa-c9b2df44923c/crucible/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7
432024-07-29T22:46:16.555ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_04fcb670-76ba-4fb0-bafa-c9b2df44923c/crucible/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7@fdbce13f-1958-413b-b274-4950ad3f94e7
442024-07-29T22:46:16.555ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_04fcb670-76ba-4fb0-bafa-c9b2df44923c/crucible/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7", "fdbce13f-1958-413b-b274-4950ad3f94e7"]
452024-07-29T22:46:16.564ZINFOcrucible-agent (datafile): requesting running snapshot ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7-fdbce13f-1958-413b-b274-4950ad3f94e7 state: Requested
462024-07-29T22:46:16.565ZINFOcrucible-agent (worker): applying SMF actions for region ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7 running snapshot fdbce13f-1958-413b-b274-4950ad3f94e7 (state Requested)...
472024-07-29T22:46:16.565ZINFOcrucible-agent (dropshot): request completed latency_us = 27833 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36775 req_id = aea238bb-793e-4297-a08f-f889c92c4411 response_code = 200 uri = /crucible/0/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7/snapshots/fdbce13f-1958-413b-b274-4950ad3f94e7/run
482024-07-29T22:46:16.566ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7
492024-07-29T22:46:16.566ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
502024-07-29T22:46:16.566ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::17
512024-07-29T22:46:16.566ZINFOcrucible-agent (worker): reconfig required
522024-07-29T22:46:16.566ZINFOcrucible-agent (worker): using existing config property group
532024-07-29T22:46:16.567ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7
542024-07-29T22:46:16.567ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7
552024-07-29T22:46:16.567ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
562024-07-29T22:46:16.567ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
572024-07-29T22:46:16.567ZINFOcrucible-agent (worker): commit
582024-07-29T22:46:16.572ZINFOcrucible-agent (worker): ok!
592024-07-29T22:46:16.572ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7-fdbce13f-1958-413b-b274-4950ad3f94e7
602024-07-29T22:46:16.578ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7-fdbce13f-1958-413b-b274-4950ad3f94e7
612024-07-29T22:46:16.589ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7-fdbce13f-1958-413b-b274-4950ad3f94e7
622024-07-29T22:46:16.589ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7/.zfs/snapshot/fdbce13f-1958-413b-b274-4950ad3f94e7
632024-07-29T22:46:16.589ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
642024-07-29T22:46:16.589ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
652024-07-29T22:46:16.590ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
662024-07-29T22:46:16.590ZINFOcrucible-agent (worker): commit
672024-07-29T22:46:16.593ZINFOcrucible-agent (worker): ok!
682024-07-29T22:46:16.599ZINFOcrucible-agent (worker): SMF ok!
692024-07-29T22:46:16.599ZINFOcrucible-agent (datafile): region ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7 running snapshot fdbce13f-1958-413b-b274-4950ad3f94e7 state: Requested -> Created
702024-07-29T22:46:19.346ZINFOcrucible-agent (datafile): region 36892883-61b2-40d0-bebf-c245441f57f1 state: Requested
712024-07-29T22:46:19.346ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
722024-07-29T22:46:19.346ZINFOcrucible-agent (dropshot): request completed latency_us = 853 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:59948 req_id = fb62b40b-7e41-4d77-8b41-ebcbbea17a94 response_code = 200 uri = /crucible/0/regions
732024-07-29T22:46:19.355ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_04fcb670-76ba-4fb0-bafa-c9b2df44923c/crucible/regions/36892883-61b2-40d0-bebf-c245441f57f1
742024-07-29T22:46:19.355ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_04fcb670-76ba-4fb0-bafa-c9b2df44923c/crucible/regions/36892883-61b2-40d0-bebf-c245441f57f1
752024-07-29T22:46:19.403ZINFOcrucible-agent (worker): creating region Region { id: RegionId("36892883-61b2-40d0-bebf-c245441f57f1"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/36892883-61b2-40d0-bebf-c245441f57f1" region = 36892883-61b2-40d0-bebf-c245441f57f1
762024-07-29T22:46:19.403ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "36892883-61b2-40d0-bebf-c245441f57f1" "--data" "/data/regions/36892883-61b2-40d0-bebf-c245441f57f1" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 36892883-61b2-40d0-bebf-c245441f57f1
772024-07-29T22:46:19.473ZINFOcrucible-agent (worker): region files created ok region = 36892883-61b2-40d0-bebf-c245441f57f1
782024-07-29T22:46:19.473ZINFOcrucible-agent (datafile): region 36892883-61b2-40d0-bebf-c245441f57f1 state: Requested -> Created
792024-07-29T22:46:19.473ZINFOcrucible-agent (worker): applying SMF actions post create...
802024-07-29T22:46:19.473ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-36892883-61b2-40d0-bebf-c245441f57f1
812024-07-29T22:46:19.477ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-36892883-61b2-40d0-bebf-c245441f57f1
822024-07-29T22:46:19.484ZINFOcrucible-agent (worker): creating config property group
832024-07-29T22:46:19.487ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-36892883-61b2-40d0-bebf-c245441f57f1
842024-07-29T22:46:19.487ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/36892883-61b2-40d0-bebf-c245441f57f1
852024-07-29T22:46:19.488ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
862024-07-29T22:46:19.488ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
872024-07-29T22:46:19.488ZINFOcrucible-agent (worker): commit
882024-07-29T22:46:19.491ZINFOcrucible-agent (worker): ok!
892024-07-29T22:46:19.497ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7
902024-07-29T22:46:19.497ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
912024-07-29T22:46:19.497ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::17
922024-07-29T22:46:19.497ZINFOcrucible-agent (worker): reconfig required
932024-07-29T22:46:19.497ZINFOcrucible-agent (worker): using existing config property group
942024-07-29T22:46:19.497ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7
952024-07-29T22:46:19.499ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:59948 req_id = 58eeced6-8a5c-4fd3-bab1-6f1b9d61fb38 response_code = 200 uri = /crucible/0/regions
962024-07-29T22:46:19.500ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7
972024-07-29T22:46:19.500ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
982024-07-29T22:46:19.500ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
992024-07-29T22:46:19.500ZINFOcrucible-agent (worker): commit
1002024-07-29T22:46:19.504ZINFOcrucible-agent (worker): ok!
1012024-07-29T22:46:19.506ZINFOcrucible-agent (worker): SMF ok!
1022024-07-29T22:46:49.510ZINFOcrucible-agent (dropshot): request completed latency_us = 64 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:59948 req_id = 7fec843a-5008-47d7-a6ae-56c5e28f48b3 response_code = 200 uri = /crucible/0/regions/36892883-61b2-40d0-bebf-c245441f57f1
1032024-07-29T22:46:49.513ZINFOcrucible-agent (datafile): path is "/data/regions/36892883-61b2-40d0-bebf-c245441f57f1"
1042024-07-29T22:46:49.524ZINFOcrucible-agent (datafile): dataset is oxp_04fcb670-76ba-4fb0-bafa-c9b2df44923c/crucible/regions/36892883-61b2-40d0-bebf-c245441f57f1
1052024-07-29T22:46:49.532ZINFOcrucible-agent (datafile): region 36892883-61b2-40d0-bebf-c245441f57f1 state: Created -> Tombstoned
1062024-07-29T22:46:49.532ZINFOcrucible-agent (worker): applying SMF actions before removal...
1072024-07-29T22:46:49.532ZINFOcrucible-agent (dropshot): request completed latency_us = 19429 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:59948 req_id = c14a2e71-f743-4489-b159-c12ffabc1e27 response_code = 204 uri = /crucible/0/regions/36892883-61b2-40d0-bebf-c245441f57f1
1082024-07-29T22:46:49.534ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-36892883-61b2-40d0-bebf-c245441f57f1 (instance states: (Some(Online), None))
1092024-07-29T22:46:49.536ZINFOcrucible-agent (dropshot): request completed latency_us = 594 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:59948 req_id = 4af8b7f4-a009-4024-87e7-d00a5c42af75 response_code = 200 uri = /crucible/0/regions/36892883-61b2-40d0-bebf-c245441f57f1
1102024-07-29T22:46:49.545ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7
1112024-07-29T22:46:49.545ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1122024-07-29T22:46:49.545ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::17
1132024-07-29T22:46:49.545ZINFOcrucible-agent (worker): reconfig required
1142024-07-29T22:46:49.545ZINFOcrucible-agent (worker): using existing config property group
1152024-07-29T22:46:49.545ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7
1162024-07-29T22:46:49.546ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ec5b87a2-3b2d-43af-9f82-5ceeb6b55eb7
1172024-07-29T22:46:49.546ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1182024-07-29T22:46:49.546ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
1192024-07-29T22:46:49.546ZINFOcrucible-agent (worker): commit
1202024-07-29T22:46:49.554ZINFOcrucible-agent (worker): ok!
1212024-07-29T22:46:49.556ZINFOcrucible-agent (worker): SMF ok!
1222024-07-29T22:46:49.565ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_04fcb670-76ba-4fb0-bafa-c9b2df44923c/crucible/regions/36892883-61b2-40d0-bebf-c245441f57f1" region = 36892883-61b2-40d0-bebf-c245441f57f1
1232024-07-29T22:46:49.616ZINFOcrucible-agent (datafile): region 36892883-61b2-40d0-bebf-c245441f57f1 state: Tombstoned -> Destroyed
1242024-07-29T22:46:49.669ZINFOcrucible-agent (dropshot): request completed latency_us = 630 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:59948 req_id = efb6a884-9ff2-4b27-b39d-c215ef96496d response_code = 200 uri = /crucible/0/regions/36892883-61b2-40d0-bebf-c245441f57f1