Max level shown:
12024-07-04T19:31:12.730ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:49133
22024-07-04T19:31:12.731ZINFOcrucible-agent (datafile): region 00f56126-684f-42ae-b025-efb1e5875df8 state: Requested
32024-07-04T19:31:12.731ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-07-04T19:31:12.731ZINFOcrucible-agent (dropshot): request completed latency_us = 1178 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49133 req_id = ac4a2fdd-1889-4afa-a56b-3ba5e749757d response_code = 200 uri = /crucible/0/regions
52024-07-04T19:31:12.744ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_9633ee26-fe12-4383-92d4-5ffe245edc8f/crucible/regions/00f56126-684f-42ae-b025-efb1e5875df8
62024-07-04T19:31:12.744ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_9633ee26-fe12-4383-92d4-5ffe245edc8f/crucible/regions/00f56126-684f-42ae-b025-efb1e5875df8
72024-07-04T19:31:12.805ZINFOcrucible-agent (worker): creating region Region { id: RegionId("00f56126-684f-42ae-b025-efb1e5875df8"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/00f56126-684f-42ae-b025-efb1e5875df8" region = 00f56126-684f-42ae-b025-efb1e5875df8
82024-07-04T19:31:12.805ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "00f56126-684f-42ae-b025-efb1e5875df8" "--data" "/data/regions/00f56126-684f-42ae-b025-efb1e5875df8" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 00f56126-684f-42ae-b025-efb1e5875df8
92024-07-04T19:31:12.955ZINFOcrucible-agent (worker): region files created ok region = 00f56126-684f-42ae-b025-efb1e5875df8
102024-07-04T19:31:12.955ZINFOcrucible-agent (datafile): region 00f56126-684f-42ae-b025-efb1e5875df8 state: Requested -> Created
112024-07-04T19:31:12.955ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-07-04T19:31:12.956ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-00f56126-684f-42ae-b025-efb1e5875df8
132024-07-04T19:31:12.964ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-00f56126-684f-42ae-b025-efb1e5875df8
142024-07-04T19:31:12.983ZINFOcrucible-agent (worker): creating config property group
152024-07-04T19:31:12.988ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-00f56126-684f-42ae-b025-efb1e5875df8
162024-07-04T19:31:12.989ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/00f56126-684f-42ae-b025-efb1e5875df8
172024-07-04T19:31:12.989ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182024-07-04T19:31:12.989ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
192024-07-04T19:31:12.989ZINFOcrucible-agent (worker): commit
202024-07-04T19:31:12.993ZINFOcrucible-agent (worker): ok!
212024-07-04T19:31:13.001ZINFOcrucible-agent (worker): SMF ok!
222024-07-04T19:31:13.084ZINFOcrucible-agent (dropshot): request completed latency_us = 94 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49133 req_id = 609c6abc-6196-4966-b535-e1ae294e5dc0 response_code = 200 uri = /crucible/0/regions
232024-07-04T19:31:14.289ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:33799
242024-07-04T19:31:14.289ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:33799 req_id = 83684aa8-193a-42a7-9f7b-6b458a65dfeb response_code = 200 uri = /crucible/0/regions/9c7860bc-23b6-4c4d-9d34-5cb2834aad1d
252024-07-04T19:31:14.289ZINFOcrucible-agent (datafile): path is "/data/regions/9c7860bc-23b6-4c4d-9d34-5cb2834aad1d"
262024-07-04T19:31:14.298ZINFOcrucible-agent (datafile): dataset is oxp_9633ee26-fe12-4383-92d4-5ffe245edc8f/crucible/regions/9c7860bc-23b6-4c4d-9d34-5cb2834aad1d
272024-07-04T19:31:14.306ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_9633ee26-fe12-4383-92d4-5ffe245edc8f/crucible/regions/9c7860bc-23b6-4c4d-9d34-5cb2834aad1d@e40f4dd5-23cb-4b2b-840a-5686b36dcf05
282024-07-04T19:31:14.306ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_9633ee26-fe12-4383-92d4-5ffe245edc8f/crucible/regions/9c7860bc-23b6-4c4d-9d34-5cb2834aad1d", "e40f4dd5-23cb-4b2b-840a-5686b36dcf05"]
292024-07-04T19:31:14.317ZINFOcrucible-agent (dropshot): request completed latency_us = 28149 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:33799 req_id = 67cfb55e-04c9-450a-b394-41c251414dfb response_code = 200 uri = /crucible/0/regions/9c7860bc-23b6-4c4d-9d34-5cb2834aad1d/snapshots/e40f4dd5-23cb-4b2b-840a-5686b36dcf05
302024-07-04T19:31:14.318ZINFOcrucible-agent (datafile): path is "/data/regions/9c7860bc-23b6-4c4d-9d34-5cb2834aad1d"
312024-07-04T19:31:14.327ZINFOcrucible-agent (datafile): dataset is oxp_9633ee26-fe12-4383-92d4-5ffe245edc8f/crucible/regions/9c7860bc-23b6-4c4d-9d34-5cb2834aad1d
322024-07-04T19:31:14.335ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_9633ee26-fe12-4383-92d4-5ffe245edc8f/crucible/regions/9c7860bc-23b6-4c4d-9d34-5cb2834aad1d@e40f4dd5-23cb-4b2b-840a-5686b36dcf05
332024-07-04T19:31:14.335ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_9633ee26-fe12-4383-92d4-5ffe245edc8f/crucible/regions/9c7860bc-23b6-4c4d-9d34-5cb2834aad1d", "e40f4dd5-23cb-4b2b-840a-5686b36dcf05"]
342024-07-04T19:31:14.347ZINFOcrucible-agent (datafile): requesting running snapshot 9c7860bc-23b6-4c4d-9d34-5cb2834aad1d-e40f4dd5-23cb-4b2b-840a-5686b36dcf05 state: Requested
352024-07-04T19:31:14.347ZINFOcrucible-agent (worker): applying SMF actions for region 9c7860bc-23b6-4c4d-9d34-5cb2834aad1d running snapshot e40f4dd5-23cb-4b2b-840a-5686b36dcf05 (state Requested)...
362024-07-04T19:31:14.347ZINFOcrucible-agent (dropshot): request completed latency_us = 29569 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33799 req_id = 558ee903-dc66-4d98-8506-b5696f625f1d response_code = 200 uri = /crucible/0/regions/9c7860bc-23b6-4c4d-9d34-5cb2834aad1d/snapshots/e40f4dd5-23cb-4b2b-840a-5686b36dcf05/run
372024-07-04T19:31:14.350ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-9c7860bc-23b6-4c4d-9d34-5cb2834aad1d-e40f4dd5-23cb-4b2b-840a-5686b36dcf05
382024-07-04T19:31:14.359ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-9c7860bc-23b6-4c4d-9d34-5cb2834aad1d-e40f4dd5-23cb-4b2b-840a-5686b36dcf05
392024-07-04T19:31:14.377ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-9c7860bc-23b6-4c4d-9d34-5cb2834aad1d-e40f4dd5-23cb-4b2b-840a-5686b36dcf05
402024-07-04T19:31:14.377ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9c7860bc-23b6-4c4d-9d34-5cb2834aad1d/.zfs/snapshot/e40f4dd5-23cb-4b2b-840a-5686b36dcf05
412024-07-04T19:31:14.377ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
422024-07-04T19:31:14.377ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
432024-07-04T19:31:14.377ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
442024-07-04T19:31:14.377ZINFOcrucible-agent (worker): commit
452024-07-04T19:31:14.384ZINFOcrucible-agent (worker): ok!
462024-07-04T19:31:14.391ZINFOcrucible-agent (worker): SMF ok!
472024-07-04T19:31:14.391ZINFOcrucible-agent (datafile): region 9c7860bc-23b6-4c4d-9d34-5cb2834aad1d running snapshot e40f4dd5-23cb-4b2b-840a-5686b36dcf05 state: Requested -> Created
482024-07-04T19:31:17.526ZINFOcrucible-agent (datafile): region a7663159-a452-4d0e-9216-f24608c523c8 state: Requested
492024-07-04T19:31:17.526ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
502024-07-04T19:31:17.526ZINFOcrucible-agent (dropshot): request completed latency_us = 197 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49133 req_id = e32ad0cc-2f2d-472f-a269-c40f988287a9 response_code = 200 uri = /crucible/0/regions
512024-07-04T19:31:17.536ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_9633ee26-fe12-4383-92d4-5ffe245edc8f/crucible/regions/a7663159-a452-4d0e-9216-f24608c523c8
522024-07-04T19:31:17.536ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_9633ee26-fe12-4383-92d4-5ffe245edc8f/crucible/regions/a7663159-a452-4d0e-9216-f24608c523c8
532024-07-04T19:31:17.609ZINFOcrucible-agent (worker): creating region Region { id: RegionId("a7663159-a452-4d0e-9216-f24608c523c8"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/a7663159-a452-4d0e-9216-f24608c523c8" region = a7663159-a452-4d0e-9216-f24608c523c8
542024-07-04T19:31:17.609ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "a7663159-a452-4d0e-9216-f24608c523c8" "--data" "/data/regions/a7663159-a452-4d0e-9216-f24608c523c8" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = a7663159-a452-4d0e-9216-f24608c523c8
552024-07-04T19:31:17.721ZINFOcrucible-agent (worker): region files created ok region = a7663159-a452-4d0e-9216-f24608c523c8
562024-07-04T19:31:17.721ZINFOcrucible-agent (datafile): region a7663159-a452-4d0e-9216-f24608c523c8 state: Requested -> Created
572024-07-04T19:31:17.721ZINFOcrucible-agent (worker): applying SMF actions post create...
582024-07-04T19:31:17.724ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-a7663159-a452-4d0e-9216-f24608c523c8
592024-07-04T19:31:17.730ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-a7663159-a452-4d0e-9216-f24608c523c8
602024-07-04T19:31:17.746ZINFOcrucible-agent (worker): creating config property group
612024-07-04T19:31:17.752ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-a7663159-a452-4d0e-9216-f24608c523c8
622024-07-04T19:31:17.753ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a7663159-a452-4d0e-9216-f24608c523c8
632024-07-04T19:31:17.753ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
642024-07-04T19:31:17.753ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
652024-07-04T19:31:17.753ZINFOcrucible-agent (worker): commit
662024-07-04T19:31:17.763ZINFOcrucible-agent (worker): ok!
672024-07-04T19:31:17.775ZINFOcrucible-agent (worker): SMF ok!
682024-07-04T19:31:17.832ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49133 req_id = db5130f3-3b84-4e9a-8b9d-9994df487360 response_code = 200 uri = /crucible/0/regions
692024-07-04T19:31:42.650ZINFOcrucible-agent (dropshot): request completed latency_us = 49 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:49133 req_id = 77842b88-8e90-4a80-b3a5-bbf05f5aa0d7 response_code = 200 uri = /crucible/0/regions/a7663159-a452-4d0e-9216-f24608c523c8
702024-07-04T19:31:42.654ZINFOcrucible-agent (datafile): path is "/data/regions/a7663159-a452-4d0e-9216-f24608c523c8"
712024-07-04T19:31:42.664ZINFOcrucible-agent (datafile): dataset is oxp_9633ee26-fe12-4383-92d4-5ffe245edc8f/crucible/regions/a7663159-a452-4d0e-9216-f24608c523c8
722024-07-04T19:31:42.671ZINFOcrucible-agent (datafile): region a7663159-a452-4d0e-9216-f24608c523c8 state: Created -> Tombstoned
732024-07-04T19:31:42.672ZINFOcrucible-agent (worker): applying SMF actions before removal...
742024-07-04T19:31:42.672ZINFOcrucible-agent (dropshot): request completed latency_us = 18567 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:49133 req_id = df810029-3b4e-4eaa-b8c2-f47728f56797 response_code = 204 uri = /crucible/0/regions/a7663159-a452-4d0e-9216-f24608c523c8
752024-07-04T19:31:42.673ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-a7663159-a452-4d0e-9216-f24608c523c8 (instance states: (Some(Online), None))
762024-07-04T19:31:42.675ZINFOcrucible-agent (dropshot): request completed latency_us = 25 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:49133 req_id = d4ad3ef9-d43a-43f4-82ad-f72147995ba2 response_code = 200 uri = /crucible/0/regions/a7663159-a452-4d0e-9216-f24608c523c8
772024-07-04T19:31:42.680ZINFOcrucible-agent (worker): SMF ok!
782024-07-04T19:31:42.690ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_9633ee26-fe12-4383-92d4-5ffe245edc8f/crucible/regions/a7663159-a452-4d0e-9216-f24608c523c8" region = a7663159-a452-4d0e-9216-f24608c523c8
792024-07-04T19:31:42.735ZINFOcrucible-agent (datafile): region a7663159-a452-4d0e-9216-f24608c523c8 state: Tombstoned -> Destroyed
802024-07-04T19:31:42.798ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:49133 req_id = 1a77f9dc-dad2-4b48-8950-dcfa315825f3 response_code = 200 uri = /crucible/0/regions/a7663159-a452-4d0e-9216-f24608c523c8