Max level shown:
12024-06-20T21:34:55.801ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:52752
22024-06-20T21:34:55.805ZINFOcrucible-agent (dropshot): request completed latency_us = 3433 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:52752 req_id = 51a8aa2e-2d46-4493-bd7a-d73e2e11702d response_code = 200 uri = /crucible/0/regions/6ff590b1-72a7-43b8-bd91-2839666aa782
32024-06-20T21:34:55.805ZINFOcrucible-agent (datafile): path is "/data/regions/6ff590b1-72a7-43b8-bd91-2839666aa782"
42024-06-20T21:34:55.814ZINFOcrucible-agent (datafile): dataset is oxp_68941de4-9d96-43c2-b5bb-c76900595397/crucible/regions/6ff590b1-72a7-43b8-bd91-2839666aa782
52024-06-20T21:34:55.823ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_68941de4-9d96-43c2-b5bb-c76900595397/crucible/regions/6ff590b1-72a7-43b8-bd91-2839666aa782@198c4152-c4bb-4411-985a-4952d2ee4af1
62024-06-20T21:34:55.823ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_68941de4-9d96-43c2-b5bb-c76900595397/crucible/regions/6ff590b1-72a7-43b8-bd91-2839666aa782", "198c4152-c4bb-4411-985a-4952d2ee4af1"]
72024-06-20T21:34:55.836ZINFOcrucible-agent (dropshot): request completed latency_us = 31423 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:52752 req_id = b42a82c1-3d3f-463c-aef6-ce651f604f1f response_code = 200 uri = /crucible/0/regions/6ff590b1-72a7-43b8-bd91-2839666aa782/snapshots/198c4152-c4bb-4411-985a-4952d2ee4af1
82024-06-20T21:34:55.836ZINFOcrucible-agent (datafile): path is "/data/regions/6ff590b1-72a7-43b8-bd91-2839666aa782"
92024-06-20T21:34:55.845ZINFOcrucible-agent (datafile): dataset is oxp_68941de4-9d96-43c2-b5bb-c76900595397/crucible/regions/6ff590b1-72a7-43b8-bd91-2839666aa782
102024-06-20T21:34:55.854ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_68941de4-9d96-43c2-b5bb-c76900595397/crucible/regions/6ff590b1-72a7-43b8-bd91-2839666aa782@198c4152-c4bb-4411-985a-4952d2ee4af1
112024-06-20T21:34:55.854ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_68941de4-9d96-43c2-b5bb-c76900595397/crucible/regions/6ff590b1-72a7-43b8-bd91-2839666aa782", "198c4152-c4bb-4411-985a-4952d2ee4af1"]
122024-06-20T21:34:55.864ZINFOcrucible-agent (datafile): requesting running snapshot 6ff590b1-72a7-43b8-bd91-2839666aa782-198c4152-c4bb-4411-985a-4952d2ee4af1 state: Requested
132024-06-20T21:34:55.865ZINFOcrucible-agent (worker): applying SMF actions for region 6ff590b1-72a7-43b8-bd91-2839666aa782 running snapshot 198c4152-c4bb-4411-985a-4952d2ee4af1 (state Requested)...
142024-06-20T21:34:55.865ZINFOcrucible-agent (dropshot): request completed latency_us = 28417 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52752 req_id = 73227fb3-1599-4b53-9c77-2ff4cedd6697 response_code = 200 uri = /crucible/0/regions/6ff590b1-72a7-43b8-bd91-2839666aa782/snapshots/198c4152-c4bb-4411-985a-4952d2ee4af1/run
152024-06-20T21:34:55.871ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/6ff590b1-72a7-43b8-bd91-2839666aa782
162024-06-20T21:34:55.871ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
172024-06-20T21:34:55.871ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
182024-06-20T21:34:55.871ZINFOcrucible-agent (worker): reconfig required
192024-06-20T21:34:55.871ZINFOcrucible-agent (worker): using existing config property group
202024-06-20T21:34:55.871ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6ff590b1-72a7-43b8-bd91-2839666aa782
212024-06-20T21:34:55.871ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6ff590b1-72a7-43b8-bd91-2839666aa782
222024-06-20T21:34:55.871ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
232024-06-20T21:34:55.871ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
242024-06-20T21:34:55.872ZINFOcrucible-agent (worker): commit
252024-06-20T21:34:55.876ZINFOcrucible-agent (worker): ok!
262024-06-20T21:34:55.876ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-6ff590b1-72a7-43b8-bd91-2839666aa782-198c4152-c4bb-4411-985a-4952d2ee4af1
272024-06-20T21:34:55.879ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-6ff590b1-72a7-43b8-bd91-2839666aa782-198c4152-c4bb-4411-985a-4952d2ee4af1
282024-06-20T21:34:55.892ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-6ff590b1-72a7-43b8-bd91-2839666aa782-198c4152-c4bb-4411-985a-4952d2ee4af1
292024-06-20T21:34:55.892ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6ff590b1-72a7-43b8-bd91-2839666aa782/.zfs/snapshot/198c4152-c4bb-4411-985a-4952d2ee4af1
302024-06-20T21:34:55.892ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
312024-06-20T21:34:55.892ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
322024-06-20T21:34:55.892ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
332024-06-20T21:34:55.893ZINFOcrucible-agent (worker): commit
342024-06-20T21:34:55.896ZINFOcrucible-agent (worker): ok!
352024-06-20T21:34:55.900ZINFOcrucible-agent (worker): SMF ok!
362024-06-20T21:34:55.900ZINFOcrucible-agent (datafile): region 6ff590b1-72a7-43b8-bd91-2839666aa782 running snapshot 198c4152-c4bb-4411-985a-4952d2ee4af1 state: Requested -> Created
372024-06-20T21:34:58.632ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:55822
382024-06-20T21:34:58.632ZINFOcrucible-agent (datafile): region 90e33b38-f94f-4069-9db8-1d80687d5ba4 state: Requested
392024-06-20T21:34:58.632ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
402024-06-20T21:34:58.632ZINFOcrucible-agent (dropshot): request completed latency_us = 450 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:55822 req_id = 07e0fcc6-bb18-4b3e-8b20-ab640dfd0450 response_code = 200 uri = /crucible/0/regions
412024-06-20T21:34:58.643ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_68941de4-9d96-43c2-b5bb-c76900595397/crucible/regions/90e33b38-f94f-4069-9db8-1d80687d5ba4
422024-06-20T21:34:58.643ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_68941de4-9d96-43c2-b5bb-c76900595397/crucible/regions/90e33b38-f94f-4069-9db8-1d80687d5ba4
432024-06-20T21:34:58.710ZINFOcrucible-agent (worker): creating region Region { id: RegionId("90e33b38-f94f-4069-9db8-1d80687d5ba4"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/90e33b38-f94f-4069-9db8-1d80687d5ba4" region = 90e33b38-f94f-4069-9db8-1d80687d5ba4
442024-06-20T21:34:58.712ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "90e33b38-f94f-4069-9db8-1d80687d5ba4" "--data" "/data/regions/90e33b38-f94f-4069-9db8-1d80687d5ba4" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 90e33b38-f94f-4069-9db8-1d80687d5ba4
452024-06-20T21:34:58.764ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:55822 req_id = 243c5f4f-b2c2-404b-a0cd-ec42d121d0bc response_code = 200 uri = /crucible/0/regions
462024-06-20T21:34:58.832ZINFOcrucible-agent (worker): region files created ok region = 90e33b38-f94f-4069-9db8-1d80687d5ba4
472024-06-20T21:34:58.832ZINFOcrucible-agent (datafile): region 90e33b38-f94f-4069-9db8-1d80687d5ba4 state: Requested -> Created
482024-06-20T21:34:58.832ZINFOcrucible-agent (worker): applying SMF actions post create...
492024-06-20T21:34:58.833ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/6ff590b1-72a7-43b8-bd91-2839666aa782
502024-06-20T21:34:58.833ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
512024-06-20T21:34:58.833ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
522024-06-20T21:34:58.833ZINFOcrucible-agent (worker): reconfig required
532024-06-20T21:34:58.833ZINFOcrucible-agent (worker): using existing config property group
542024-06-20T21:34:58.833ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6ff590b1-72a7-43b8-bd91-2839666aa782
552024-06-20T21:34:58.834ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6ff590b1-72a7-43b8-bd91-2839666aa782
562024-06-20T21:34:58.834ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
572024-06-20T21:34:58.834ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
582024-06-20T21:34:58.834ZINFOcrucible-agent (worker): commit
592024-06-20T21:34:58.837ZINFOcrucible-agent (worker): ok!
602024-06-20T21:34:58.838ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-90e33b38-f94f-4069-9db8-1d80687d5ba4
612024-06-20T21:34:58.844ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-90e33b38-f94f-4069-9db8-1d80687d5ba4
622024-06-20T21:34:58.852ZINFOcrucible-agent (worker): creating config property group
632024-06-20T21:34:58.855ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-90e33b38-f94f-4069-9db8-1d80687d5ba4
642024-06-20T21:34:58.855ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/90e33b38-f94f-4069-9db8-1d80687d5ba4
652024-06-20T21:34:58.855ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
662024-06-20T21:34:58.855ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
672024-06-20T21:34:58.855ZINFOcrucible-agent (worker): commit
682024-06-20T21:34:58.861ZINFOcrucible-agent (worker): ok!
692024-06-20T21:34:58.867ZINFOcrucible-agent (worker): SMF ok!
702024-06-20T21:34:59.119ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:55822 req_id = 398082cc-4bc3-47c9-8648-4142ee5450bb response_code = 200 uri = /crucible/0/regions
712024-06-20T21:35:25.880ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:55822 req_id = 0a60897f-6274-4f2f-86f6-cad9d170e0eb response_code = 200 uri = /crucible/0/regions/90e33b38-f94f-4069-9db8-1d80687d5ba4
722024-06-20T21:35:25.883ZINFOcrucible-agent (datafile): path is "/data/regions/90e33b38-f94f-4069-9db8-1d80687d5ba4"
732024-06-20T21:35:25.896ZINFOcrucible-agent (datafile): dataset is oxp_68941de4-9d96-43c2-b5bb-c76900595397/crucible/regions/90e33b38-f94f-4069-9db8-1d80687d5ba4
742024-06-20T21:35:25.904ZINFOcrucible-agent (datafile): region 90e33b38-f94f-4069-9db8-1d80687d5ba4 state: Created -> Tombstoned
752024-06-20T21:35:25.904ZINFOcrucible-agent (worker): applying SMF actions before removal...
762024-06-20T21:35:25.907ZINFOcrucible-agent (dropshot): request completed latency_us = 23500 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:55822 req_id = e1a55991-9e15-49bb-adf1-a677b2235cfa response_code = 204 uri = /crucible/0/regions/90e33b38-f94f-4069-9db8-1d80687d5ba4
772024-06-20T21:35:25.908ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-90e33b38-f94f-4069-9db8-1d80687d5ba4 (instance states: (Some(Online), None))
782024-06-20T21:35:25.910ZINFOcrucible-agent (dropshot): request completed latency_us = 29 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:55822 req_id = ac87234d-b063-46cd-92fa-b36c3e4c909b response_code = 200 uri = /crucible/0/regions/90e33b38-f94f-4069-9db8-1d80687d5ba4
792024-06-20T21:35:25.916ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/6ff590b1-72a7-43b8-bd91-2839666aa782
802024-06-20T21:35:25.916ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
812024-06-20T21:35:25.916ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
822024-06-20T21:35:25.916ZINFOcrucible-agent (worker): reconfig required
832024-06-20T21:35:25.916ZINFOcrucible-agent (worker): using existing config property group
842024-06-20T21:35:25.916ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6ff590b1-72a7-43b8-bd91-2839666aa782
852024-06-20T21:35:25.917ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6ff590b1-72a7-43b8-bd91-2839666aa782
862024-06-20T21:35:25.917ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
872024-06-20T21:35:25.917ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
882024-06-20T21:35:25.917ZINFOcrucible-agent (worker): commit
892024-06-20T21:35:25.925ZINFOcrucible-agent (worker): ok!
902024-06-20T21:35:25.926ZINFOcrucible-agent (worker): SMF ok!
912024-06-20T21:35:25.937ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_68941de4-9d96-43c2-b5bb-c76900595397/crucible/regions/90e33b38-f94f-4069-9db8-1d80687d5ba4" region = 90e33b38-f94f-4069-9db8-1d80687d5ba4
922024-06-20T21:35:25.998ZINFOcrucible-agent (datafile): region 90e33b38-f94f-4069-9db8-1d80687d5ba4 state: Tombstoned -> Destroyed
932024-06-20T21:35:26.031ZINFOcrucible-agent (dropshot): request completed latency_us = 79 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:55822 req_id = fa498137-055a-4d11-aa9b-58f6bf686cea response_code = 200 uri = /crucible/0/regions/90e33b38-f94f-4069-9db8-1d80687d5ba4