Max level shown:
1 [ Aug 22 02:17:51 Enabled. ]
2 [ Aug 22 02:17:51 Rereading configuration. ]
3 [ Aug 22 02:17:53 Rereading configuration. ]
4 [ Aug 22 02:17:56 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible -l [fd00:1122:3344:101::12]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-08-22T02:17:56.316ZINFOcrucible-agent: dataset: "oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible"
72024-08-22T02:17:56.317ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::12]:32345
82024-08-22T02:17:56.317ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-08-22T02:17:56.335ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-08-22T02:17:56.513ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::12]:32345
112024-08-22T02:18:19.013ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:60946
122024-08-22T02:18:19.028ZINFOcrucible-agent (datafile): region bd57ff32-8011-48ff-8db3-59788b6d9f11 state: Requested
132024-08-22T02:18:19.029ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
142024-08-22T02:18:19.029ZINFOcrucible-agent (dropshot): request completed latency_us = 7922 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60946 req_id = 7be6a345-44aa-44ba-aa17-b554d6535353 response_code = 200 uri = /crucible/0/regions
152024-08-22T02:18:19.041ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/bd57ff32-8011-48ff-8db3-59788b6d9f11
162024-08-22T02:18:19.041ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/bd57ff32-8011-48ff-8db3-59788b6d9f11
172024-08-22T02:18:19.105ZINFOcrucible-agent (worker): creating region Region { id: RegionId("bd57ff32-8011-48ff-8db3-59788b6d9f11"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/bd57ff32-8011-48ff-8db3-59788b6d9f11" region = bd57ff32-8011-48ff-8db3-59788b6d9f11
182024-08-22T02:18:19.107ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "bd57ff32-8011-48ff-8db3-59788b6d9f11" "--data" "/data/regions/bd57ff32-8011-48ff-8db3-59788b6d9f11" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = bd57ff32-8011-48ff-8db3-59788b6d9f11
192024-08-22T02:18:19.222ZINFOcrucible-agent (worker): region files created ok region = bd57ff32-8011-48ff-8db3-59788b6d9f11
202024-08-22T02:18:19.222ZINFOcrucible-agent (datafile): region bd57ff32-8011-48ff-8db3-59788b6d9f11 state: Requested -> Created
212024-08-22T02:18:19.223ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-08-22T02:18:19.223ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-bd57ff32-8011-48ff-8db3-59788b6d9f11
232024-08-22T02:18:19.227ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-bd57ff32-8011-48ff-8db3-59788b6d9f11
242024-08-22T02:18:19.235ZINFOcrucible-agent (worker): creating config property group
252024-08-22T02:18:19.238ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-bd57ff32-8011-48ff-8db3-59788b6d9f11
262024-08-22T02:18:19.245ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/bd57ff32-8011-48ff-8db3-59788b6d9f11
272024-08-22T02:18:19.245ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-08-22T02:18:19.245ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
292024-08-22T02:18:19.245ZINFOcrucible-agent (worker): commit
302024-08-22T02:18:19.249ZINFOcrucible-agent (worker): ok!
312024-08-22T02:18:19.254ZINFOcrucible-agent (worker): SMF ok!
322024-08-22T02:18:19.387ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60946 req_id = cca72aa2-a0bf-4a8b-9fd0-a5a40521450f response_code = 200 uri = /crucible/0/regions
332024-08-22T02:18:20.311ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:60946 req_id = 5ee8f549-cbd4-47ac-b663-9912bd6026c2 response_code = 200 uri = /crucible/0/regions/bd57ff32-8011-48ff-8db3-59788b6d9f11
342024-08-22T02:18:20.313ZINFOcrucible-agent (datafile): path is "/data/regions/bd57ff32-8011-48ff-8db3-59788b6d9f11"
352024-08-22T02:18:20.322ZINFOcrucible-agent (datafile): dataset is oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/bd57ff32-8011-48ff-8db3-59788b6d9f11
362024-08-22T02:18:20.331ZINFOcrucible-agent (datafile): region bd57ff32-8011-48ff-8db3-59788b6d9f11 state: Created -> Tombstoned
372024-08-22T02:18:20.331ZINFOcrucible-agent (worker): applying SMF actions before removal...
382024-08-22T02:18:20.331ZINFOcrucible-agent (dropshot): request completed latency_us = 17712 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:60946 req_id = a50234dc-5cd5-44df-9d46-4a54fc69639f response_code = 204 uri = /crucible/0/regions/bd57ff32-8011-48ff-8db3-59788b6d9f11
392024-08-22T02:18:20.332ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-bd57ff32-8011-48ff-8db3-59788b6d9f11 (instance states: (Some(Online), None))
402024-08-22T02:18:20.334ZINFOcrucible-agent (dropshot): request completed latency_us = 23 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:60946 req_id = 92585a51-7bd7-4864-95a1-5220e4389c46 response_code = 200 uri = /crucible/0/regions/bd57ff32-8011-48ff-8db3-59788b6d9f11
412024-08-22T02:18:20.336ZINFOcrucible-agent (worker): SMF ok!
422024-08-22T02:18:20.346ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/bd57ff32-8011-48ff-8db3-59788b6d9f11" region = bd57ff32-8011-48ff-8db3-59788b6d9f11
432024-08-22T02:18:20.400ZINFOcrucible-agent (datafile): region bd57ff32-8011-48ff-8db3-59788b6d9f11 state: Tombstoned -> Destroyed
442024-08-22T02:18:20.463ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:60946 req_id = b0fdc92f-db62-421e-a24b-acb6ed281435 response_code = 200 uri = /crucible/0/regions/bd57ff32-8011-48ff-8db3-59788b6d9f11
452024-08-22T02:18:21.652ZINFOcrucible-agent (datafile): region 6774a237-5391-41e7-9ba1-99c0d9bc247c state: Requested
462024-08-22T02:18:21.652ZINFOcrucible-agent (dropshot): request completed latency_us = 205 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60946 req_id = 3904897a-24db-4cf0-a954-363c0fd447ad response_code = 200 uri = /crucible/0/regions
472024-08-22T02:18:21.652ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
482024-08-22T02:18:21.661ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c
492024-08-22T02:18:21.661ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c
502024-08-22T02:18:21.712ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6774a237-5391-41e7-9ba1-99c0d9bc247c"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c" region = 6774a237-5391-41e7-9ba1-99c0d9bc247c
512024-08-22T02:18:21.714ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "6774a237-5391-41e7-9ba1-99c0d9bc247c" "--data" "/data/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 6774a237-5391-41e7-9ba1-99c0d9bc247c
522024-08-22T02:18:21.785ZINFOcrucible-agent (worker): region files created ok region = 6774a237-5391-41e7-9ba1-99c0d9bc247c
532024-08-22T02:18:21.785ZINFOcrucible-agent (datafile): region 6774a237-5391-41e7-9ba1-99c0d9bc247c state: Requested -> Created
542024-08-22T02:18:21.785ZINFOcrucible-agent (worker): applying SMF actions post create...
552024-08-22T02:18:21.787ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6774a237-5391-41e7-9ba1-99c0d9bc247c
562024-08-22T02:18:21.790ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6774a237-5391-41e7-9ba1-99c0d9bc247c
572024-08-22T02:18:21.798ZINFOcrucible-agent (worker): creating config property group
582024-08-22T02:18:21.802ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6774a237-5391-41e7-9ba1-99c0d9bc247c
592024-08-22T02:18:21.802ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c
602024-08-22T02:18:21.802ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
612024-08-22T02:18:21.802ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
622024-08-22T02:18:21.802ZINFOcrucible-agent (worker): commit
632024-08-22T02:18:21.805ZINFOcrucible-agent (worker): ok!
642024-08-22T02:18:21.813ZINFOcrucible-agent (worker): SMF ok!
652024-08-22T02:18:21.991ZINFOcrucible-agent (dropshot): request completed latency_us = 82 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:60946 req_id = cb3a2649-1bf6-4173-a36c-c28f5d819f0e response_code = 200 uri = /crucible/0/regions
662024-08-22T02:23:38.242ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:47082
672024-08-22T02:23:38.242ZINFOcrucible-agent (datafile): region e05b52d6-732e-40a5-90c8-500acae0f758 state: Requested
682024-08-22T02:23:38.243ZINFOcrucible-agent (dropshot): request completed latency_us = 682 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:47082 req_id = 059b3514-b53f-4026-b5fc-de61a92d02fb response_code = 200 uri = /crucible/0/regions
692024-08-22T02:23:38.243ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
702024-08-22T02:23:38.253ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/e05b52d6-732e-40a5-90c8-500acae0f758
712024-08-22T02:23:38.253ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/e05b52d6-732e-40a5-90c8-500acae0f758
722024-08-22T02:23:38.307ZINFOcrucible-agent (worker): creating region Region { id: RegionId("e05b52d6-732e-40a5-90c8-500acae0f758"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/e05b52d6-732e-40a5-90c8-500acae0f758" region = e05b52d6-732e-40a5-90c8-500acae0f758
732024-08-22T02:23:38.307ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "e05b52d6-732e-40a5-90c8-500acae0f758" "--data" "/data/regions/e05b52d6-732e-40a5-90c8-500acae0f758" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = e05b52d6-732e-40a5-90c8-500acae0f758
742024-08-22T02:23:38.376ZINFOcrucible-agent (worker): region files created ok region = e05b52d6-732e-40a5-90c8-500acae0f758
752024-08-22T02:23:38.376ZINFOcrucible-agent (datafile): region e05b52d6-732e-40a5-90c8-500acae0f758 state: Requested -> Created
762024-08-22T02:23:38.376ZINFOcrucible-agent (worker): applying SMF actions post create...
772024-08-22T02:23:38.384ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-e05b52d6-732e-40a5-90c8-500acae0f758
782024-08-22T02:23:38.390ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-e05b52d6-732e-40a5-90c8-500acae0f758
792024-08-22T02:23:38.401ZINFOcrucible-agent (worker): creating config property group
802024-08-22T02:23:38.403ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-e05b52d6-732e-40a5-90c8-500acae0f758
812024-08-22T02:23:38.404ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e05b52d6-732e-40a5-90c8-500acae0f758
822024-08-22T02:23:38.404ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
832024-08-22T02:23:38.404ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
842024-08-22T02:23:38.404ZINFOcrucible-agent (worker): commit
852024-08-22T02:23:38.407ZINFOcrucible-agent (worker): ok!
862024-08-22T02:23:38.410ZINFOcrucible-agent (dropshot): request completed latency_us = 80 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:47082 req_id = 6d72682f-28f3-4fdc-a996-3a418f719b29 response_code = 200 uri = /crucible/0/regions
872024-08-22T02:23:38.412ZINFOcrucible-agent (worker): SMF ok!
882024-08-22T02:23:39.160ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:47082 req_id = 2ef596e5-da84-4571-be42-0c737af2cafc response_code = 200 uri = /crucible/0/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c
892024-08-22T02:23:39.165ZINFOcrucible-agent (datafile): path is "/data/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c"
902024-08-22T02:23:39.175ZINFOcrucible-agent (datafile): dataset is oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c
912024-08-22T02:23:39.184ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c@7278170f-9fb3-4726-884b-5c6e9eedfa7b
922024-08-22T02:23:39.184ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c", "7278170f-9fb3-4726-884b-5c6e9eedfa7b"]
932024-08-22T02:23:39.194ZINFOcrucible-agent (dropshot): request completed latency_us = 31369 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:47082 req_id = 9e63e18d-14e2-4842-b04d-fe6e9229ca61 response_code = 200 uri = /crucible/0/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c/snapshots/7278170f-9fb3-4726-884b-5c6e9eedfa7b
942024-08-22T02:23:39.198ZINFOcrucible-agent (datafile): path is "/data/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c"
952024-08-22T02:23:39.207ZINFOcrucible-agent (datafile): dataset is oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c
962024-08-22T02:23:39.215ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c@7278170f-9fb3-4726-884b-5c6e9eedfa7b
972024-08-22T02:23:39.215ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c", "7278170f-9fb3-4726-884b-5c6e9eedfa7b"]
982024-08-22T02:23:39.225ZINFOcrucible-agent (datafile): requesting running snapshot 6774a237-5391-41e7-9ba1-99c0d9bc247c-7278170f-9fb3-4726-884b-5c6e9eedfa7b state: Requested
992024-08-22T02:23:39.225ZINFOcrucible-agent (worker): applying SMF actions for region 6774a237-5391-41e7-9ba1-99c0d9bc247c running snapshot 7278170f-9fb3-4726-884b-5c6e9eedfa7b (state Requested)...
1002024-08-22T02:23:39.225ZINFOcrucible-agent (dropshot): request completed latency_us = 27481 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:47082 req_id = 1b6a3a0e-a87e-494e-8f4a-9322358b9549 response_code = 200 uri = /crucible/0/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c/snapshots/7278170f-9fb3-4726-884b-5c6e9eedfa7b/run
1012024-08-22T02:23:39.228ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-6774a237-5391-41e7-9ba1-99c0d9bc247c-7278170f-9fb3-4726-884b-5c6e9eedfa7b
1022024-08-22T02:23:39.231ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-6774a237-5391-41e7-9ba1-99c0d9bc247c-7278170f-9fb3-4726-884b-5c6e9eedfa7b
1032024-08-22T02:23:39.241ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-6774a237-5391-41e7-9ba1-99c0d9bc247c-7278170f-9fb3-4726-884b-5c6e9eedfa7b
1042024-08-22T02:23:39.241ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c/.zfs/snapshot/7278170f-9fb3-4726-884b-5c6e9eedfa7b
1052024-08-22T02:23:39.241ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1062024-08-22T02:23:39.241ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
1072024-08-22T02:23:39.241ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
1082024-08-22T02:23:39.241ZINFOcrucible-agent (worker): commit
1092024-08-22T02:23:39.245ZINFOcrucible-agent (worker): ok!
1102024-08-22T02:23:39.249ZINFOcrucible-agent (worker): SMF ok!
1112024-08-22T02:23:39.249ZINFOcrucible-agent (datafile): region 6774a237-5391-41e7-9ba1-99c0d9bc247c running snapshot 7278170f-9fb3-4726-884b-5c6e9eedfa7b state: Requested -> Created
1122024-08-22T02:23:39.592ZINFOcrucible-agent (datafile): path is "/data/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c"
1132024-08-22T02:23:39.601ZINFOcrucible-agent (datafile): dataset is oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c
1142024-08-22T02:23:39.609ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c@7278170f-9fb3-4726-884b-5c6e9eedfa7b
1152024-08-22T02:23:39.609ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c", "7278170f-9fb3-4726-884b-5c6e9eedfa7b"]
1162024-08-22T02:23:39.617ZINFOcrucible-agent (dropshot): request completed latency_us = 25009 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:47082 req_id = dfbd3290-82d6-4c7b-858d-5c7a3db473b4 response_code = 200 uri = /crucible/0/regions/6774a237-5391-41e7-9ba1-99c0d9bc247c/snapshots/7278170f-9fb3-4726-884b-5c6e9eedfa7b/run
1172024-08-22T02:23:43.116ZINFOcrucible-agent (datafile): region 92c40a6c-e056-416e-ae77-51c4f61f2bd6 state: Requested
1182024-08-22T02:23:43.116ZINFOcrucible-agent (dropshot): request completed latency_us = 232 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:47082 req_id = c45029c0-0b82-418f-9908-62913924d63c response_code = 200 uri = /crucible/0/regions
1192024-08-22T02:23:43.116ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
1202024-08-22T02:23:43.127ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/92c40a6c-e056-416e-ae77-51c4f61f2bd6
1212024-08-22T02:23:43.127ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/92c40a6c-e056-416e-ae77-51c4f61f2bd6
1222024-08-22T02:23:43.198ZINFOcrucible-agent (worker): creating region Region { id: RegionId("92c40a6c-e056-416e-ae77-51c4f61f2bd6"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/92c40a6c-e056-416e-ae77-51c4f61f2bd6" region = 92c40a6c-e056-416e-ae77-51c4f61f2bd6
1232024-08-22T02:23:43.201ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "92c40a6c-e056-416e-ae77-51c4f61f2bd6" "--data" "/data/regions/92c40a6c-e056-416e-ae77-51c4f61f2bd6" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 92c40a6c-e056-416e-ae77-51c4f61f2bd6
1242024-08-22T02:23:43.268ZINFOcrucible-agent (worker): region files created ok region = 92c40a6c-e056-416e-ae77-51c4f61f2bd6
1252024-08-22T02:23:43.268ZINFOcrucible-agent (datafile): region 92c40a6c-e056-416e-ae77-51c4f61f2bd6 state: Requested -> Created
1262024-08-22T02:23:43.268ZINFOcrucible-agent (worker): applying SMF actions post create...
1272024-08-22T02:23:43.270ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-92c40a6c-e056-416e-ae77-51c4f61f2bd6
1282024-08-22T02:23:43.273ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-92c40a6c-e056-416e-ae77-51c4f61f2bd6
1292024-08-22T02:23:43.280ZINFOcrucible-agent (worker): creating config property group
1302024-08-22T02:23:43.283ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-92c40a6c-e056-416e-ae77-51c4f61f2bd6
1312024-08-22T02:23:43.284ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/92c40a6c-e056-416e-ae77-51c4f61f2bd6
1322024-08-22T02:23:43.284ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
1332024-08-22T02:23:43.284ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
1342024-08-22T02:23:43.284ZINFOcrucible-agent (worker): commit
1352024-08-22T02:23:43.288ZINFOcrucible-agent (worker): ok!
1362024-08-22T02:23:43.299ZINFOcrucible-agent (worker): SMF ok!
1372024-08-22T02:23:43.427ZINFOcrucible-agent (dropshot): request completed latency_us = 66 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:47082 req_id = 61050b46-e10c-4efc-b3fe-a8eb31a94716 response_code = 200 uri = /crucible/0/regions
1382024-08-22T02:24:15.612ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:47082 req_id = d28e7690-6a2a-4045-b17b-061168ad56d7 response_code = 200 uri = /crucible/0/regions/92c40a6c-e056-416e-ae77-51c4f61f2bd6
1392024-08-22T02:24:15.614ZINFOcrucible-agent (datafile): path is "/data/regions/92c40a6c-e056-416e-ae77-51c4f61f2bd6"
1402024-08-22T02:24:15.627ZINFOcrucible-agent (datafile): dataset is oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/92c40a6c-e056-416e-ae77-51c4f61f2bd6
1412024-08-22T02:24:15.636ZINFOcrucible-agent (datafile): region 92c40a6c-e056-416e-ae77-51c4f61f2bd6 state: Created -> Tombstoned
1422024-08-22T02:24:15.636ZINFOcrucible-agent (worker): applying SMF actions before removal...
1432024-08-22T02:24:15.636ZINFOcrucible-agent (dropshot): request completed latency_us = 21741 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:47082 req_id = 0ca05d38-e965-432a-938a-6f5b39a05a5b response_code = 204 uri = /crucible/0/regions/92c40a6c-e056-416e-ae77-51c4f61f2bd6
1442024-08-22T02:24:15.638ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-92c40a6c-e056-416e-ae77-51c4f61f2bd6 (instance states: (Some(Online), None))
1452024-08-22T02:24:15.639ZINFOcrucible-agent (dropshot): request completed latency_us = 24 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:47082 req_id = 3d98bf41-808f-441f-9ac4-3b5d6b0c50cd response_code = 200 uri = /crucible/0/regions/92c40a6c-e056-416e-ae77-51c4f61f2bd6
1462024-08-22T02:24:15.645ZINFOcrucible-agent (worker): SMF ok!
1472024-08-22T02:24:15.654ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_a3847786-8a2e-45ee-87a5-075f35fc7eea/crucible/regions/92c40a6c-e056-416e-ae77-51c4f61f2bd6" region = 92c40a6c-e056-416e-ae77-51c4f61f2bd6
1482024-08-22T02:24:15.702ZINFOcrucible-agent (datafile): region 92c40a6c-e056-416e-ae77-51c4f61f2bd6 state: Tombstoned -> Destroyed
1492024-08-22T02:24:15.703ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:47082 req_id = 06c93fd9-ea82-4ac8-abb6-4dce1a2ca958 response_code = 200 uri = /crucible/0/regions/92c40a6c-e056-416e-ae77-51c4f61f2bd6