Max level shown:
1 [ Nov 18 21:00:52 Enabled. ]
2 [ Nov 18 21:00:52 Rereading configuration. ]
3 [ Nov 18 21:00:54 Rereading configuration. ]
4 [ Nov 18 21:00:59 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_28b058a3-a4b5-473e-8580-f5c606b6d30b/crucible -l [fd00:1122:3344:101::17]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-11-18T21:00:59.338ZINFOcrucible-agent: dataset: "oxp_28b058a3-a4b5-473e-8580-f5c606b6d30b/crucible"
72024-11-18T21:00:59.338ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::17]:32345
82024-11-18T21:00:59.338ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-11-18T21:00:59.355ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-11-18T21:00:59.429ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::17]:32345
112024-11-18T21:02:17.748ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:56195
122024-11-18T21:02:17.767ZINFOcrucible-agent (datafile): region 364a05f3-95d2-4e9e-8b8e-6b3f889224e8 state: Requested
132024-11-18T21:02:17.769ZINFOcrucible-agent (dropshot): request completed latency_us = 10589 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56195 req_id = 0511672e-f930-4f83-a78f-7cc35e97e91a response_code = 200 uri = /crucible/0/regions
142024-11-18T21:02:17.771ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152024-11-18T21:02:17.782ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_28b058a3-a4b5-473e-8580-f5c606b6d30b/crucible/regions/364a05f3-95d2-4e9e-8b8e-6b3f889224e8
162024-11-18T21:02:17.782ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_28b058a3-a4b5-473e-8580-f5c606b6d30b/crucible/regions/364a05f3-95d2-4e9e-8b8e-6b3f889224e8
172024-11-18T21:02:17.836ZINFOcrucible-agent (worker): creating region Region { id: RegionId("364a05f3-95d2-4e9e-8b8e-6b3f889224e8"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/364a05f3-95d2-4e9e-8b8e-6b3f889224e8" region = 364a05f3-95d2-4e9e-8b8e-6b3f889224e8
182024-11-18T21:02:17.838ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "364a05f3-95d2-4e9e-8b8e-6b3f889224e8" "--data" "/data/regions/364a05f3-95d2-4e9e-8b8e-6b3f889224e8" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 364a05f3-95d2-4e9e-8b8e-6b3f889224e8
192024-11-18T21:02:17.958ZINFOcrucible-agent (worker): region files created ok region = 364a05f3-95d2-4e9e-8b8e-6b3f889224e8
202024-11-18T21:02:17.958ZINFOcrucible-agent (datafile): region 364a05f3-95d2-4e9e-8b8e-6b3f889224e8 state: Requested -> Created
212024-11-18T21:02:17.958ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-11-18T21:02:17.958ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-364a05f3-95d2-4e9e-8b8e-6b3f889224e8
232024-11-18T21:02:17.964ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-364a05f3-95d2-4e9e-8b8e-6b3f889224e8
242024-11-18T21:02:17.966ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56195 req_id = e3182961-5043-445c-9a7e-62adc261193f response_code = 200 uri = /crucible/0/regions
252024-11-18T21:02:17.982ZINFOcrucible-agent (worker): creating config property group
262024-11-18T21:02:17.985ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-364a05f3-95d2-4e9e-8b8e-6b3f889224e8
272024-11-18T21:02:17.986ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/364a05f3-95d2-4e9e-8b8e-6b3f889224e8
282024-11-18T21:02:17.988ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-11-18T21:02:17.988ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
302024-11-18T21:02:17.988ZINFOcrucible-agent (worker): commit
312024-11-18T21:02:17.997ZINFOcrucible-agent (worker): ok!
322024-11-18T21:02:18.002ZINFOcrucible-agent (worker): SMF ok!
332024-11-18T21:02:18.343ZINFOcrucible-agent (dropshot): request completed latency_us = 96 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:56195 req_id = 305e9361-73e1-4067-ac52-d97d1e3b0813 response_code = 200 uri = /crucible/0/regions/364a05f3-95d2-4e9e-8b8e-6b3f889224e8
342024-11-18T21:02:18.346ZINFOcrucible-agent (datafile): path is "/data/regions/364a05f3-95d2-4e9e-8b8e-6b3f889224e8"
352024-11-18T21:02:18.358ZINFOcrucible-agent (datafile): dataset is oxp_28b058a3-a4b5-473e-8580-f5c606b6d30b/crucible/regions/364a05f3-95d2-4e9e-8b8e-6b3f889224e8
362024-11-18T21:02:18.368ZINFOcrucible-agent (datafile): region 364a05f3-95d2-4e9e-8b8e-6b3f889224e8 state: Created -> Tombstoned
372024-11-18T21:02:18.368ZINFOcrucible-agent (worker): applying SMF actions before removal...
382024-11-18T21:02:18.368ZINFOcrucible-agent (dropshot): request completed latency_us = 21732 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:56195 req_id = f84234e2-76a2-417d-b509-8d0ee8538d75 response_code = 204 uri = /crucible/0/regions/364a05f3-95d2-4e9e-8b8e-6b3f889224e8
392024-11-18T21:02:18.369ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-364a05f3-95d2-4e9e-8b8e-6b3f889224e8 (instance states: (Some(Online), None))
402024-11-18T21:02:18.371ZINFOcrucible-agent (dropshot): request completed latency_us = 46 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:56195 req_id = d1ace443-eaa7-4b6e-bbbf-3d5ef223873a response_code = 200 uri = /crucible/0/regions/364a05f3-95d2-4e9e-8b8e-6b3f889224e8
412024-11-18T21:02:18.374ZINFOcrucible-agent (worker): SMF ok!
422024-11-18T21:02:18.386ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_28b058a3-a4b5-473e-8580-f5c606b6d30b/crucible/regions/364a05f3-95d2-4e9e-8b8e-6b3f889224e8" region = 364a05f3-95d2-4e9e-8b8e-6b3f889224e8
432024-11-18T21:02:18.448ZINFOcrucible-agent (datafile): region 364a05f3-95d2-4e9e-8b8e-6b3f889224e8 state: Tombstoned -> Destroyed
442024-11-18T21:02:18.487ZINFOcrucible-agent (dropshot): request completed latency_us = 82 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:56195 req_id = 0cee6eab-af49-4f6f-80e9-24da97ed6cdf response_code = 200 uri = /crucible/0/regions/364a05f3-95d2-4e9e-8b8e-6b3f889224e8
452024-11-18T21:12:14.854ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:64461
462024-11-18T21:12:14.854ZINFOcrucible-agent (datafile): region 78db66de-ae42-40c5-b4f1-ba7ed6457721 state: Requested
472024-11-18T21:12:14.854ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
482024-11-18T21:12:14.854ZINFOcrucible-agent (dropshot): request completed latency_us = 640 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64461 req_id = b77ecb85-cd38-4578-bd8b-8ac16f74244e response_code = 200 uri = /crucible/0/regions
492024-11-18T21:12:14.868ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_28b058a3-a4b5-473e-8580-f5c606b6d30b/crucible/regions/78db66de-ae42-40c5-b4f1-ba7ed6457721
502024-11-18T21:12:14.868ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_28b058a3-a4b5-473e-8580-f5c606b6d30b/crucible/regions/78db66de-ae42-40c5-b4f1-ba7ed6457721
512024-11-18T21:12:14.919ZINFOcrucible-agent (worker): creating region Region { id: RegionId("78db66de-ae42-40c5-b4f1-ba7ed6457721"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/78db66de-ae42-40c5-b4f1-ba7ed6457721" region = 78db66de-ae42-40c5-b4f1-ba7ed6457721
522024-11-18T21:12:14.919ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "78db66de-ae42-40c5-b4f1-ba7ed6457721" "--data" "/data/regions/78db66de-ae42-40c5-b4f1-ba7ed6457721" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 78db66de-ae42-40c5-b4f1-ba7ed6457721
532024-11-18T21:12:14.996ZINFOcrucible-agent (worker): region files created ok region = 78db66de-ae42-40c5-b4f1-ba7ed6457721
542024-11-18T21:12:14.996ZINFOcrucible-agent (datafile): region 78db66de-ae42-40c5-b4f1-ba7ed6457721 state: Requested -> Created
552024-11-18T21:12:14.996ZINFOcrucible-agent (worker): applying SMF actions post create...
562024-11-18T21:12:14.997ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-78db66de-ae42-40c5-b4f1-ba7ed6457721
572024-11-18T21:12:15.002ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-78db66de-ae42-40c5-b4f1-ba7ed6457721
582024-11-18T21:12:15.017ZINFOcrucible-agent (worker): creating config property group
592024-11-18T21:12:15.021ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-78db66de-ae42-40c5-b4f1-ba7ed6457721
602024-11-18T21:12:15.021ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/78db66de-ae42-40c5-b4f1-ba7ed6457721
612024-11-18T21:12:15.021ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
622024-11-18T21:12:15.021ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
632024-11-18T21:12:15.021ZINFOcrucible-agent (worker): commit
642024-11-18T21:12:15.025ZINFOcrucible-agent (worker): ok!
652024-11-18T21:12:15.026ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64461 req_id = b998eaee-82a7-4b11-ab85-413956301534 response_code = 200 uri = /crucible/0/regions
662024-11-18T21:12:15.031ZINFOcrucible-agent (worker): SMF ok!
672024-11-18T21:12:20.029ZINFOcrucible-agent (datafile): region 9b9663a4-460a-47ca-8f0e-c507c1672f5b state: Requested
682024-11-18T21:12:20.029ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
692024-11-18T21:12:20.029ZINFOcrucible-agent (dropshot): request completed latency_us = 223 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64461 req_id = b076834b-8647-4e8a-9eca-55ccfec026c0 response_code = 200 uri = /crucible/0/regions
702024-11-18T21:12:20.039ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_28b058a3-a4b5-473e-8580-f5c606b6d30b/crucible/regions/9b9663a4-460a-47ca-8f0e-c507c1672f5b
712024-11-18T21:12:20.039ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_28b058a3-a4b5-473e-8580-f5c606b6d30b/crucible/regions/9b9663a4-460a-47ca-8f0e-c507c1672f5b
722024-11-18T21:12:20.092ZINFOcrucible-agent (worker): creating region Region { id: RegionId("9b9663a4-460a-47ca-8f0e-c507c1672f5b"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/9b9663a4-460a-47ca-8f0e-c507c1672f5b" region = 9b9663a4-460a-47ca-8f0e-c507c1672f5b
732024-11-18T21:12:20.094ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "9b9663a4-460a-47ca-8f0e-c507c1672f5b" "--data" "/data/regions/9b9663a4-460a-47ca-8f0e-c507c1672f5b" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 9b9663a4-460a-47ca-8f0e-c507c1672f5b
742024-11-18T21:12:20.166ZINFOcrucible-agent (worker): region files created ok region = 9b9663a4-460a-47ca-8f0e-c507c1672f5b
752024-11-18T21:12:20.166ZINFOcrucible-agent (datafile): region 9b9663a4-460a-47ca-8f0e-c507c1672f5b state: Requested -> Created
762024-11-18T21:12:20.166ZINFOcrucible-agent (worker): applying SMF actions post create...
772024-11-18T21:12:20.168ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-9b9663a4-460a-47ca-8f0e-c507c1672f5b
782024-11-18T21:12:20.172ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-9b9663a4-460a-47ca-8f0e-c507c1672f5b
792024-11-18T21:12:20.181ZINFOcrucible-agent (worker): creating config property group
802024-11-18T21:12:20.184ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-9b9663a4-460a-47ca-8f0e-c507c1672f5b
812024-11-18T21:12:20.185ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9b9663a4-460a-47ca-8f0e-c507c1672f5b
822024-11-18T21:12:20.185ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
832024-11-18T21:12:20.185ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
842024-11-18T21:12:20.185ZINFOcrucible-agent (worker): commit
852024-11-18T21:12:20.189ZINFOcrucible-agent (worker): ok!
862024-11-18T21:12:20.194ZINFOcrucible-agent (worker): SMF ok!
872024-11-18T21:12:20.221ZINFOcrucible-agent (dropshot): request completed latency_us = 737 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64461 req_id = da10cc98-4105-4e5d-b902-058513389a46 response_code = 200 uri = /crucible/0/regions
882024-11-18T21:12:45.649ZINFOcrucible-agent (dropshot): request completed latency_us = 49 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64461 req_id = f9b914c4-1030-46a7-8469-350532a8694e response_code = 200 uri = /crucible/0/regions/9b9663a4-460a-47ca-8f0e-c507c1672f5b
892024-11-18T21:12:45.653ZINFOcrucible-agent (datafile): path is "/data/regions/9b9663a4-460a-47ca-8f0e-c507c1672f5b"
902024-11-18T21:12:45.665ZINFOcrucible-agent (datafile): dataset is oxp_28b058a3-a4b5-473e-8580-f5c606b6d30b/crucible/regions/9b9663a4-460a-47ca-8f0e-c507c1672f5b
912024-11-18T21:12:45.675ZINFOcrucible-agent (datafile): region 9b9663a4-460a-47ca-8f0e-c507c1672f5b state: Created -> Tombstoned
922024-11-18T21:12:45.675ZINFOcrucible-agent (worker): applying SMF actions before removal...
932024-11-18T21:12:45.675ZINFOcrucible-agent (dropshot): request completed latency_us = 22297 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:64461 req_id = f671ad80-b89a-4a94-8be7-679922357b1c response_code = 204 uri = /crucible/0/regions/9b9663a4-460a-47ca-8f0e-c507c1672f5b
942024-11-18T21:12:45.677ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-9b9663a4-460a-47ca-8f0e-c507c1672f5b (instance states: (Some(Online), None))
952024-11-18T21:12:45.678ZINFOcrucible-agent (dropshot): request completed latency_us = 32 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64461 req_id = 5d2592b4-ae03-4666-bc46-d163350a7474 response_code = 200 uri = /crucible/0/regions/9b9663a4-460a-47ca-8f0e-c507c1672f5b
962024-11-18T21:12:45.684ZINFOcrucible-agent (worker): SMF ok!
972024-11-18T21:12:45.696ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_28b058a3-a4b5-473e-8580-f5c606b6d30b/crucible/regions/9b9663a4-460a-47ca-8f0e-c507c1672f5b" region = 9b9663a4-460a-47ca-8f0e-c507c1672f5b
982024-11-18T21:12:45.752ZINFOcrucible-agent (dropshot): request completed latency_us = 762 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64461 req_id = a723875a-1257-4b9d-87bc-08ea7de618ab response_code = 200 uri = /crucible/0/regions/9b9663a4-460a-47ca-8f0e-c507c1672f5b
992024-11-18T21:12:45.755ZINFOcrucible-agent (datafile): region 9b9663a4-460a-47ca-8f0e-c507c1672f5b state: Tombstoned -> Destroyed
1002024-11-18T21:12:45.917ZINFOcrucible-agent (dropshot): request completed latency_us = 44 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64461 req_id = 1e689958-2cdf-434b-8bed-850f341bf298 response_code = 200 uri = /crucible/0/regions/9b9663a4-460a-47ca-8f0e-c507c1672f5b