Max level shown:
12024-02-01T09:00:33.908ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:57037
22024-02-01T09:00:33.940ZINFOcrucible-agent (datafile): region 4516ea2b-d914-4c40-b86c-e70fd0818eae state: Requested
32024-02-01T09:00:33.943ZINFOcrucible-agent (dropshot): request completed latency_us = 2869 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57037 req_id = 9c76b305-2c61-4b19-b32d-2a4698fdae93 response_code = 200 uri = /crucible/0/regions
42024-02-01T09:00:33.943ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
52024-02-01T09:00:33.952ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae
62024-02-01T09:00:33.952ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae
72024-02-01T09:00:34.002ZINFOcrucible-agent (worker): creating region Region { id: RegionId("4516ea2b-d914-4c40-b86c-e70fd0818eae"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae" region = 4516ea2b-d914-4c40-b86c-e70fd0818eae
82024-02-01T09:00:34.130ZINFOcrucible-agent (worker): region files created ok region = 4516ea2b-d914-4c40-b86c-e70fd0818eae
92024-02-01T09:00:34.130ZINFOcrucible-agent (datafile): region 4516ea2b-d914-4c40-b86c-e70fd0818eae state: Requested -> Created
102024-02-01T09:00:34.131ZINFOcrucible-agent (worker): applying SMF actions post create...
112024-02-01T09:00:34.132ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-4516ea2b-d914-4c40-b86c-e70fd0818eae
122024-02-01T09:00:34.139ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-4516ea2b-d914-4c40-b86c-e70fd0818eae
132024-02-01T09:00:34.150ZINFOcrucible-agent (dropshot): request completed latency_us = 79 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57037 req_id = d73d913e-9e00-4a51-9c23-7366ead618f3 response_code = 200 uri = /crucible/0/regions
142024-02-01T09:00:34.157ZINFOcrucible-agent (worker): creating config property group
152024-02-01T09:00:34.161ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4516ea2b-d914-4c40-b86c-e70fd0818eae
162024-02-01T09:00:34.161ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae
172024-02-01T09:00:34.161ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
182024-02-01T09:00:34.161ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
192024-02-01T09:00:34.161ZINFOcrucible-agent (worker): commit
202024-02-01T09:00:34.168ZINFOcrucible-agent (worker): ok!
212024-02-01T09:00:34.179ZINFOcrucible-agent (worker): SMF ok!
222024-02-01T09:05:58.299ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:35829
232024-02-01T09:05:58.299ZINFOcrucible-agent (datafile): region 8ebcd679-db0f-4a84-b34b-5d387d6d65d4 state: Requested
242024-02-01T09:05:58.300ZINFOcrucible-agent (dropshot): request completed latency_us = 776 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:35829 req_id = fdd29df8-9328-4a0a-8e08-2ba75e6c6e90 response_code = 200 uri = /crucible/0/regions
252024-02-01T09:05:58.300ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
262024-02-01T09:05:58.311ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/8ebcd679-db0f-4a84-b34b-5d387d6d65d4
272024-02-01T09:05:58.311ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/8ebcd679-db0f-4a84-b34b-5d387d6d65d4
282024-02-01T09:05:58.392ZINFOcrucible-agent (worker): creating region Region { id: RegionId("8ebcd679-db0f-4a84-b34b-5d387d6d65d4"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/8ebcd679-db0f-4a84-b34b-5d387d6d65d4" region = 8ebcd679-db0f-4a84-b34b-5d387d6d65d4
292024-02-01T09:05:58.477ZINFOcrucible-agent (dropshot): request completed latency_us = 145 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:35829 req_id = c84ca4da-1de6-4b56-a7ef-14917e1a6003 response_code = 200 uri = /crucible/0/regions
302024-02-01T09:05:58.527ZINFOcrucible-agent (worker): region files created ok region = 8ebcd679-db0f-4a84-b34b-5d387d6d65d4
312024-02-01T09:05:58.527ZINFOcrucible-agent (datafile): region 8ebcd679-db0f-4a84-b34b-5d387d6d65d4 state: Requested -> Created
322024-02-01T09:05:58.527ZINFOcrucible-agent (worker): applying SMF actions post create...
332024-02-01T09:05:58.534ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-8ebcd679-db0f-4a84-b34b-5d387d6d65d4
342024-02-01T09:05:58.541ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-8ebcd679-db0f-4a84-b34b-5d387d6d65d4
352024-02-01T09:05:58.558ZINFOcrucible-agent (worker): creating config property group
362024-02-01T09:05:58.565ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-8ebcd679-db0f-4a84-b34b-5d387d6d65d4
372024-02-01T09:05:58.566ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/8ebcd679-db0f-4a84-b34b-5d387d6d65d4
382024-02-01T09:05:58.566ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
392024-02-01T09:05:58.566ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
402024-02-01T09:05:58.566ZINFOcrucible-agent (worker): commit
412024-02-01T09:05:58.572ZINFOcrucible-agent (worker): ok!
422024-02-01T09:05:58.584ZINFOcrucible-agent (worker): SMF ok!
432024-02-01T09:05:59.116ZINFOcrucible-agent (dropshot): request completed latency_us = 80 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:35829 req_id = 847cbba9-9fcc-4154-9762-8b7076ec679c response_code = 200 uri = /crucible/0/regions
442024-02-01T09:05:59.648ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:50240
452024-02-01T09:05:59.648ZINFOcrucible-agent (dropshot): request completed latency_us = 665 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50240 req_id = 2fc1e56f-d148-4f16-bae0-cf94a6bd6efe response_code = 200 uri = /crucible/0/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae
462024-02-01T09:05:59.651ZINFOcrucible-agent (datafile): path is "/data/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae"
472024-02-01T09:05:59.659ZINFOcrucible-agent (datafile): dataset is oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae
482024-02-01T09:05:59.669ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae@2839f329-0315-4eef-84bf-18eb46bddaa5
492024-02-01T09:05:59.669ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae", "2839f329-0315-4eef-84bf-18eb46bddaa5"]
502024-02-01T09:05:59.678ZINFOcrucible-agent (dropshot): request completed latency_us = 29649 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50240 req_id = 2f86edf0-bb8a-4f11-ba6d-be37e395963a response_code = 200 uri = /crucible/0/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae/snapshots/2839f329-0315-4eef-84bf-18eb46bddaa5
512024-02-01T09:05:59.678ZINFOcrucible-agent (datafile): path is "/data/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae"
522024-02-01T09:05:59.687ZINFOcrucible-agent (datafile): dataset is oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae
532024-02-01T09:05:59.696ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae@2839f329-0315-4eef-84bf-18eb46bddaa5
542024-02-01T09:05:59.696ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae", "2839f329-0315-4eef-84bf-18eb46bddaa5"]
552024-02-01T09:05:59.706ZINFOcrucible-agent (datafile): requesting running snapshot 4516ea2b-d914-4c40-b86c-e70fd0818eae-2839f329-0315-4eef-84bf-18eb46bddaa5 state: Requested
562024-02-01T09:05:59.706ZINFOcrucible-agent (worker): applying SMF actions for region 4516ea2b-d914-4c40-b86c-e70fd0818eae running snapshot 2839f329-0315-4eef-84bf-18eb46bddaa5 (state Requested)...
572024-02-01T09:05:59.706ZINFOcrucible-agent (dropshot): request completed latency_us = 27505 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50240 req_id = 106c03bc-e585-49f7-a156-82409a6f35a7 response_code = 200 uri = /crucible/0/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae/snapshots/2839f329-0315-4eef-84bf-18eb46bddaa5/run
582024-02-01T09:05:59.708ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-4516ea2b-d914-4c40-b86c-e70fd0818eae-2839f329-0315-4eef-84bf-18eb46bddaa5
592024-02-01T09:05:59.717ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-4516ea2b-d914-4c40-b86c-e70fd0818eae-2839f329-0315-4eef-84bf-18eb46bddaa5
602024-02-01T09:05:59.734ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-4516ea2b-d914-4c40-b86c-e70fd0818eae-2839f329-0315-4eef-84bf-18eb46bddaa5
612024-02-01T09:05:59.734ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4516ea2b-d914-4c40-b86c-e70fd0818eae/.zfs/snapshot/2839f329-0315-4eef-84bf-18eb46bddaa5
622024-02-01T09:05:59.734ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
632024-02-01T09:05:59.734ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-02-01T09:05:59.734ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
652024-02-01T09:05:59.734ZINFOcrucible-agent (worker): commit
662024-02-01T09:05:59.739ZINFOcrucible-agent (worker): ok!
672024-02-01T09:05:59.748ZINFOcrucible-agent (worker): SMF ok!
682024-02-01T09:05:59.748ZINFOcrucible-agent (datafile): region 4516ea2b-d914-4c40-b86c-e70fd0818eae running snapshot 2839f329-0315-4eef-84bf-18eb46bddaa5 state: Requested -> Created
692024-02-01T09:06:02.315ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:33658
702024-02-01T09:06:02.347ZINFOcrucible-agent (datafile): region 22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7 state: Requested
712024-02-01T09:06:02.347ZINFOcrucible-agent (dropshot): request completed latency_us = 214 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33658 req_id = f4758631-ecd2-41e6-a9b2-bb9dbb6d089c response_code = 200 uri = /crucible/0/regions
722024-02-01T09:06:02.347ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
732024-02-01T09:06:02.357ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7
742024-02-01T09:06:02.357ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7
752024-02-01T09:06:02.431ZINFOcrucible-agent (worker): creating region Region { id: RegionId("22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7" region = 22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7
762024-02-01T09:06:02.478ZINFOcrucible-agent (dropshot): request completed latency_us = 82 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33658 req_id = 1a982ac8-fad1-4af4-880d-fa7e03cff3f1 response_code = 200 uri = /crucible/0/regions
772024-02-01T09:06:02.542ZINFOcrucible-agent (worker): region files created ok region = 22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7
782024-02-01T09:06:02.542ZINFOcrucible-agent (datafile): region 22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7 state: Requested -> Created
792024-02-01T09:06:02.542ZINFOcrucible-agent (worker): applying SMF actions post create...
802024-02-01T09:06:02.543ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7
812024-02-01T09:06:02.549ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7
822024-02-01T09:06:02.561ZINFOcrucible-agent (worker): creating config property group
832024-02-01T09:06:02.568ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7
842024-02-01T09:06:02.568ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7
852024-02-01T09:06:02.568ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
862024-02-01T09:06:02.568ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
872024-02-01T09:06:02.569ZINFOcrucible-agent (worker): commit
882024-02-01T09:06:02.576ZINFOcrucible-agent (worker): ok!
892024-02-01T09:06:02.593ZINFOcrucible-agent (worker): SMF ok!
902024-02-01T09:06:02.762ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33658 req_id = a188ba43-ab50-4a8d-8c34-88593e6f7a50 response_code = 200 uri = /crucible/0/regions
912024-02-01T09:06:33.597ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:39501
922024-02-01T09:06:33.597ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39501 req_id = 10b7a44c-d10b-449a-9182-d80273c8aa7e response_code = 200 uri = /crucible/0/regions/22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7
932024-02-01T09:06:33.597ZINFOcrucible-agent (datafile): path is "/data/regions/22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7"
942024-02-01T09:06:33.608ZINFOcrucible-agent (datafile): dataset is oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7
952024-02-01T09:06:33.616ZINFOcrucible-agent (datafile): region 22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7 state: Created -> Tombstoned
962024-02-01T09:06:33.617ZINFOcrucible-agent (worker): applying SMF actions before removal...
972024-02-01T09:06:33.617ZINFOcrucible-agent (dropshot): request completed latency_us = 19403 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:39501 req_id = abe14eeb-38ed-407c-8ff1-c73c9aa88a83 response_code = 204 uri = /crucible/0/regions/22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7
982024-02-01T09:06:33.617ZINFOcrucible-agent (dropshot): request completed latency_us = 25 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39501 req_id = 54b464c2-1825-4ac1-b3a0-026308d9b974 response_code = 200 uri = /crucible/0/regions/22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7
992024-02-01T09:06:33.618ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7 (instance states: (Some(Online), None))
1002024-02-01T09:06:33.628ZINFOcrucible-agent (worker): SMF ok!
1012024-02-01T09:06:33.638ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7" region = 22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7
1022024-02-01T09:06:33.683ZINFOcrucible-agent (datafile): region 22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7 state: Tombstoned -> Destroyed
1032024-02-01T09:06:33.687ZINFOcrucible-agent (dropshot): request completed latency_us = 110 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39501 req_id = 80a1a9bb-2166-4836-a669-735ab9d90a0f response_code = 200 uri = /crucible/0/regions/22e9cdca-3e63-4c37-b6fc-bc5fa4e1bee7