Max level shown:
12024-08-28T01:02:25.020ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:51140
22024-08-28T01:02:25.020ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51140 req_id = 203fab6d-b83c-4d92-8485-9ffc65633d22 response_code = 200 uri = /crucible/0/regions/66469867-092c-43a4-9229-8c7b5f674119
32024-08-28T01:02:25.028ZINFOcrucible-agent (datafile): path is "/data/regions/66469867-092c-43a4-9229-8c7b5f674119"
42024-08-28T01:02:25.038ZINFOcrucible-agent (datafile): dataset is oxp_8c0884cf-f16d-4d49-a677-c20655479168/crucible/regions/66469867-092c-43a4-9229-8c7b5f674119
52024-08-28T01:02:25.047ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_8c0884cf-f16d-4d49-a677-c20655479168/crucible/regions/66469867-092c-43a4-9229-8c7b5f674119@27978712-5943-4c98-9768-8b7ead029887
62024-08-28T01:02:25.047ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_8c0884cf-f16d-4d49-a677-c20655479168/crucible/regions/66469867-092c-43a4-9229-8c7b5f674119", "27978712-5943-4c98-9768-8b7ead029887"]
72024-08-28T01:02:25.059ZINFOcrucible-agent (dropshot): request completed latency_us = 34925 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51140 req_id = aab55853-3181-4087-a6b5-71f19dd12aac response_code = 200 uri = /crucible/0/regions/66469867-092c-43a4-9229-8c7b5f674119/snapshots/27978712-5943-4c98-9768-8b7ead029887
82024-08-28T01:02:25.063ZINFOcrucible-agent (datafile): path is "/data/regions/66469867-092c-43a4-9229-8c7b5f674119"
92024-08-28T01:02:25.071ZINFOcrucible-agent (datafile): dataset is oxp_8c0884cf-f16d-4d49-a677-c20655479168/crucible/regions/66469867-092c-43a4-9229-8c7b5f674119
102024-08-28T01:02:25.080ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_8c0884cf-f16d-4d49-a677-c20655479168/crucible/regions/66469867-092c-43a4-9229-8c7b5f674119@27978712-5943-4c98-9768-8b7ead029887
112024-08-28T01:02:25.080ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_8c0884cf-f16d-4d49-a677-c20655479168/crucible/regions/66469867-092c-43a4-9229-8c7b5f674119", "27978712-5943-4c98-9768-8b7ead029887"]
122024-08-28T01:02:25.089ZINFOcrucible-agent (datafile): requesting running snapshot 66469867-092c-43a4-9229-8c7b5f674119-27978712-5943-4c98-9768-8b7ead029887 state: Requested
132024-08-28T01:02:25.090ZINFOcrucible-agent (worker): applying SMF actions for region 66469867-092c-43a4-9229-8c7b5f674119 running snapshot 27978712-5943-4c98-9768-8b7ead029887 (state Requested)...
142024-08-28T01:02:25.090ZINFOcrucible-agent (dropshot): request completed latency_us = 27243 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51140 req_id = 3960ae33-3c83-473a-ab77-65054170af02 response_code = 200 uri = /crucible/0/regions/66469867-092c-43a4-9229-8c7b5f674119/snapshots/27978712-5943-4c98-9768-8b7ead029887/run
152024-08-28T01:02:25.097ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-66469867-092c-43a4-9229-8c7b5f674119-27978712-5943-4c98-9768-8b7ead029887
162024-08-28T01:02:25.103ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-66469867-092c-43a4-9229-8c7b5f674119-27978712-5943-4c98-9768-8b7ead029887
172024-08-28T01:02:25.123ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-66469867-092c-43a4-9229-8c7b5f674119-27978712-5943-4c98-9768-8b7ead029887
182024-08-28T01:02:25.123ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/66469867-092c-43a4-9229-8c7b5f674119/.zfs/snapshot/27978712-5943-4c98-9768-8b7ead029887
192024-08-28T01:02:25.123ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
202024-08-28T01:02:25.124ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
212024-08-28T01:02:25.124ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
222024-08-28T01:02:25.124ZINFOcrucible-agent (worker): commit
232024-08-28T01:02:25.130ZINFOcrucible-agent (worker): ok!
242024-08-28T01:02:25.135ZINFOcrucible-agent (worker): SMF ok!
252024-08-28T01:02:25.135ZINFOcrucible-agent (datafile): region 66469867-092c-43a4-9229-8c7b5f674119 running snapshot 27978712-5943-4c98-9768-8b7ead029887 state: Requested -> Created
262024-08-28T01:02:25.394ZINFOcrucible-agent (datafile): path is "/data/regions/66469867-092c-43a4-9229-8c7b5f674119"
272024-08-28T01:02:25.403ZINFOcrucible-agent (datafile): dataset is oxp_8c0884cf-f16d-4d49-a677-c20655479168/crucible/regions/66469867-092c-43a4-9229-8c7b5f674119
282024-08-28T01:02:25.411ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_8c0884cf-f16d-4d49-a677-c20655479168/crucible/regions/66469867-092c-43a4-9229-8c7b5f674119@27978712-5943-4c98-9768-8b7ead029887
292024-08-28T01:02:25.411ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_8c0884cf-f16d-4d49-a677-c20655479168/crucible/regions/66469867-092c-43a4-9229-8c7b5f674119", "27978712-5943-4c98-9768-8b7ead029887"]
302024-08-28T01:02:25.418ZINFOcrucible-agent (dropshot): request completed latency_us = 25123 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51140 req_id = c636d653-be6d-4f5f-b613-babc66a076a8 response_code = 200 uri = /crucible/0/regions/66469867-092c-43a4-9229-8c7b5f674119/snapshots/27978712-5943-4c98-9768-8b7ead029887/run
312024-08-28T01:02:29.495ZINFOcrucible-agent (datafile): region 3068b079-35a5-460f-b81e-6194fe6a65b6 state: Requested
322024-08-28T01:02:29.495ZINFOcrucible-agent (dropshot): request completed latency_us = 931 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51140 req_id = 2fbf688d-c6cd-411d-9161-bdebf7363796 response_code = 200 uri = /crucible/0/regions
332024-08-28T01:02:29.495ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
342024-08-28T01:02:29.505ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_8c0884cf-f16d-4d49-a677-c20655479168/crucible/regions/3068b079-35a5-460f-b81e-6194fe6a65b6
352024-08-28T01:02:29.505ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_8c0884cf-f16d-4d49-a677-c20655479168/crucible/regions/3068b079-35a5-460f-b81e-6194fe6a65b6
362024-08-28T01:02:29.582ZINFOcrucible-agent (worker): creating region Region { id: RegionId("3068b079-35a5-460f-b81e-6194fe6a65b6"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/3068b079-35a5-460f-b81e-6194fe6a65b6" region = 3068b079-35a5-460f-b81e-6194fe6a65b6
372024-08-28T01:02:29.582ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "3068b079-35a5-460f-b81e-6194fe6a65b6" "--data" "/data/regions/3068b079-35a5-460f-b81e-6194fe6a65b6" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 3068b079-35a5-460f-b81e-6194fe6a65b6
382024-08-28T01:02:29.692ZINFOcrucible-agent (worker): region files created ok region = 3068b079-35a5-460f-b81e-6194fe6a65b6
392024-08-28T01:02:29.692ZINFOcrucible-agent (datafile): region 3068b079-35a5-460f-b81e-6194fe6a65b6 state: Requested -> Created
402024-08-28T01:02:29.692ZINFOcrucible-agent (worker): applying SMF actions post create...
412024-08-28T01:02:29.693ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-3068b079-35a5-460f-b81e-6194fe6a65b6
422024-08-28T01:02:29.705ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-3068b079-35a5-460f-b81e-6194fe6a65b6
432024-08-28T01:02:29.723ZINFOcrucible-agent (worker): creating config property group
442024-08-28T01:02:29.730ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3068b079-35a5-460f-b81e-6194fe6a65b6
452024-08-28T01:02:29.730ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3068b079-35a5-460f-b81e-6194fe6a65b6
462024-08-28T01:02:29.730ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
472024-08-28T01:02:29.730ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
482024-08-28T01:02:29.730ZINFOcrucible-agent (worker): commit
492024-08-28T01:02:29.738ZINFOcrucible-agent (worker): ok!
502024-08-28T01:02:29.752ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51140 req_id = bb5ea744-068f-4c5a-8428-2b5c79b0e162 response_code = 200 uri = /crucible/0/regions
512024-08-28T01:02:29.756ZINFOcrucible-agent (worker): SMF ok!
522024-08-28T01:02:54.599ZINFOcrucible-agent (dropshot): request completed latency_us = 747 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51140 req_id = e33548f8-b891-4c21-a350-57719df45d36 response_code = 200 uri = /crucible/0/regions/3068b079-35a5-460f-b81e-6194fe6a65b6
532024-08-28T01:02:54.603ZINFOcrucible-agent (datafile): path is "/data/regions/3068b079-35a5-460f-b81e-6194fe6a65b6"
542024-08-28T01:02:54.615ZINFOcrucible-agent (datafile): dataset is oxp_8c0884cf-f16d-4d49-a677-c20655479168/crucible/regions/3068b079-35a5-460f-b81e-6194fe6a65b6
552024-08-28T01:02:54.624ZINFOcrucible-agent (datafile): region 3068b079-35a5-460f-b81e-6194fe6a65b6 state: Created -> Tombstoned
562024-08-28T01:02:54.624ZINFOcrucible-agent (worker): applying SMF actions before removal...
572024-08-28T01:02:54.624ZINFOcrucible-agent (dropshot): request completed latency_us = 20739 local_addr = [fd00:1122:3344:101::14]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:51140 req_id = 3fbe02b6-c6d8-40c2-a8a8-5bbb791cc282 response_code = 204 uri = /crucible/0/regions/3068b079-35a5-460f-b81e-6194fe6a65b6
582024-08-28T01:02:54.625ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-3068b079-35a5-460f-b81e-6194fe6a65b6 (instance states: (Some(Online), None))
592024-08-28T01:02:54.627ZINFOcrucible-agent (dropshot): request completed latency_us = 22 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51140 req_id = 35fa62fc-64db-44ca-acb3-ccea7f5225f2 response_code = 200 uri = /crucible/0/regions/3068b079-35a5-460f-b81e-6194fe6a65b6
602024-08-28T01:02:54.632ZINFOcrucible-agent (worker): SMF ok!
612024-08-28T01:02:54.643ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_8c0884cf-f16d-4d49-a677-c20655479168/crucible/regions/3068b079-35a5-460f-b81e-6194fe6a65b6" region = 3068b079-35a5-460f-b81e-6194fe6a65b6
622024-08-28T01:02:54.712ZINFOcrucible-agent (datafile): region 3068b079-35a5-460f-b81e-6194fe6a65b6 state: Tombstoned -> Destroyed
632024-08-28T01:02:54.771ZINFOcrucible-agent (dropshot): request completed latency_us = 684 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51140 req_id = d766dc83-a017-4956-bc6a-66556031b25f response_code = 200 uri = /crucible/0/regions/3068b079-35a5-460f-b81e-6194fe6a65b6