Max level shown:
1 [ Aug 27 01:30:40 Enabled. ]
2 [ Aug 27 01:30:40 Rereading configuration. ]
3 [ Aug 27 01:30:42 Rereading configuration. ]
4 [ Aug 27 01:30:45 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_c4fd38fc-99ba-4975-a8e6-b8bb5d9becbf/crucible -l [fd00:1122:3344:101::16]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-08-27T01:30:45.961ZINFOcrucible-agent: dataset: "oxp_c4fd38fc-99ba-4975-a8e6-b8bb5d9becbf/crucible"
72024-08-27T01:30:45.962ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::16]:32345
82024-08-27T01:30:45.962ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-08-27T01:30:45.975ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-08-27T01:30:46.100ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::16]:32345
112024-08-27T01:32:01.676ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:51917
122024-08-27T01:32:01.699ZINFOcrucible-agent (datafile): region 20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c state: Requested
132024-08-27T01:32:01.703ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
142024-08-27T01:32:01.707ZINFOcrucible-agent (dropshot): request completed latency_us = 15693 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51917 req_id = 9ff17b3f-47ec-4e87-867a-08a1e0586281 response_code = 200 uri = /crucible/0/regions
152024-08-27T01:32:01.715ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_c4fd38fc-99ba-4975-a8e6-b8bb5d9becbf/crucible/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
162024-08-27T01:32:01.715ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_c4fd38fc-99ba-4975-a8e6-b8bb5d9becbf/crucible/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
172024-08-27T01:32:01.762ZINFOcrucible-agent (worker): creating region Region { id: RegionId("20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c" region = 20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
182024-08-27T01:32:01.764ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c" "--data" "/data/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
192024-08-27T01:32:01.902ZINFOcrucible-agent (worker): region files created ok region = 20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
202024-08-27T01:32:01.902ZINFOcrucible-agent (datafile): region 20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c state: Requested -> Created
212024-08-27T01:32:01.902ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-08-27T01:32:01.902ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
232024-08-27T01:32:01.907ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
242024-08-27T01:32:01.926ZINFOcrucible-agent (worker): creating config property group
252024-08-27T01:32:01.930ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
262024-08-27T01:32:01.938ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
272024-08-27T01:32:01.940ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-08-27T01:32:01.940ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
292024-08-27T01:32:01.940ZINFOcrucible-agent (worker): commit
302024-08-27T01:32:01.945ZINFOcrucible-agent (worker): ok!
312024-08-27T01:32:01.949ZINFOcrucible-agent (worker): SMF ok!
322024-08-27T01:32:02.040ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51917 req_id = 4c64d484-97ac-4716-b8bb-8f27cdbda038 response_code = 200 uri = /crucible/0/regions
332024-08-27T01:36:59.353ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:36276
342024-08-27T01:36:59.354ZINFOcrucible-agent (datafile): region 0fcb6d8a-1011-49e1-887f-23050a50abc6 state: Requested
352024-08-27T01:36:59.355ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
362024-08-27T01:36:59.355ZINFOcrucible-agent (dropshot): request completed latency_us = 1044 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36276 req_id = 52af22da-d736-43cf-97ed-a9b2e83369ec response_code = 200 uri = /crucible/0/regions
372024-08-27T01:36:59.365ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_c4fd38fc-99ba-4975-a8e6-b8bb5d9becbf/crucible/regions/0fcb6d8a-1011-49e1-887f-23050a50abc6
382024-08-27T01:36:59.366ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_c4fd38fc-99ba-4975-a8e6-b8bb5d9becbf/crucible/regions/0fcb6d8a-1011-49e1-887f-23050a50abc6
392024-08-27T01:36:59.434ZINFOcrucible-agent (worker): creating region Region { id: RegionId("0fcb6d8a-1011-49e1-887f-23050a50abc6"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/0fcb6d8a-1011-49e1-887f-23050a50abc6" region = 0fcb6d8a-1011-49e1-887f-23050a50abc6
402024-08-27T01:36:59.434ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "0fcb6d8a-1011-49e1-887f-23050a50abc6" "--data" "/data/regions/0fcb6d8a-1011-49e1-887f-23050a50abc6" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 0fcb6d8a-1011-49e1-887f-23050a50abc6
412024-08-27T01:36:59.525ZINFOcrucible-agent (dropshot): request completed latency_us = 110 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36276 req_id = 93d06e97-f3b4-42bc-9b9b-eb81c0e6306e response_code = 200 uri = /crucible/0/regions
422024-08-27T01:36:59.541ZINFOcrucible-agent (worker): region files created ok region = 0fcb6d8a-1011-49e1-887f-23050a50abc6
432024-08-27T01:36:59.542ZINFOcrucible-agent (datafile): region 0fcb6d8a-1011-49e1-887f-23050a50abc6 state: Requested -> Created
442024-08-27T01:36:59.542ZINFOcrucible-agent (worker): applying SMF actions post create...
452024-08-27T01:36:59.542ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-0fcb6d8a-1011-49e1-887f-23050a50abc6
462024-08-27T01:36:59.549ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-0fcb6d8a-1011-49e1-887f-23050a50abc6
472024-08-27T01:36:59.574ZINFOcrucible-agent (worker): creating config property group
482024-08-27T01:36:59.582ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0fcb6d8a-1011-49e1-887f-23050a50abc6
492024-08-27T01:36:59.583ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0fcb6d8a-1011-49e1-887f-23050a50abc6
502024-08-27T01:36:59.583ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
512024-08-27T01:36:59.583ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
522024-08-27T01:36:59.583ZINFOcrucible-agent (worker): commit
532024-08-27T01:36:59.590ZINFOcrucible-agent (worker): ok!
542024-08-27T01:36:59.599ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
552024-08-27T01:36:59.599ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
562024-08-27T01:36:59.599ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::16
572024-08-27T01:36:59.599ZINFOcrucible-agent (worker): reconfig required
582024-08-27T01:36:59.599ZINFOcrucible-agent (worker): using existing config property group
592024-08-27T01:36:59.600ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
602024-08-27T01:36:59.600ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
612024-08-27T01:36:59.600ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
622024-08-27T01:36:59.600ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
632024-08-27T01:36:59.600ZINFOcrucible-agent (worker): commit
642024-08-27T01:36:59.621ZINFOcrucible-agent (worker): ok!
652024-08-27T01:36:59.622ZINFOcrucible-agent (worker): SMF ok!
662024-08-27T01:36:59.977ZINFOcrucible-agent (dropshot): request completed latency_us = 1315 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36276 req_id = ea74dc7b-11fa-461b-a658-b30213f41939 response_code = 200 uri = /crucible/0/regions
672024-08-27T01:37:00.029ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::b]:43433
682024-08-27T01:37:00.030ZINFOcrucible-agent (dropshot): request completed latency_us = 41 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:43433 req_id = a87e1a9b-64c3-4e68-b333-5e41e98cd80e response_code = 200 uri = /crucible/0/regions/0fcb6d8a-1011-49e1-887f-23050a50abc6
692024-08-27T01:37:00.108ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:52073
702024-08-27T01:37:00.108ZINFOcrucible-agent (dropshot): request completed latency_us = 29 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:52073 req_id = 8bb07760-74a4-4708-90f6-88e40fec4f28 response_code = 200 uri = /crucible/0/regions/0fcb6d8a-1011-49e1-887f-23050a50abc6
712024-08-27T01:37:00.194ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::c]:55327
722024-08-27T01:37:00.194ZINFOcrucible-agent (dropshot): request completed latency_us = 33 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:55327 req_id = b614cc8f-b1d5-460c-a0dc-f750f251ae1d response_code = 200 uri = /crucible/0/regions/0fcb6d8a-1011-49e1-887f-23050a50abc6
732024-08-27T01:37:00.711ZINFOcrucible-agent (dropshot): request completed latency_us = 44 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36276 req_id = faab6f1d-74a4-405e-947e-60d9182a6d08 response_code = 200 uri = /crucible/0/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
742024-08-27T01:37:00.716ZINFOcrucible-agent (datafile): path is "/data/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c"
752024-08-27T01:37:00.726ZINFOcrucible-agent (datafile): dataset is oxp_c4fd38fc-99ba-4975-a8e6-b8bb5d9becbf/crucible/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
762024-08-27T01:37:00.735ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_c4fd38fc-99ba-4975-a8e6-b8bb5d9becbf/crucible/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c@2f669e1e-7251-4220-ac4e-00bc54e3a4e7
772024-08-27T01:37:00.735ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_c4fd38fc-99ba-4975-a8e6-b8bb5d9becbf/crucible/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c", "2f669e1e-7251-4220-ac4e-00bc54e3a4e7"]
782024-08-27T01:37:00.746ZINFOcrucible-agent (dropshot): request completed latency_us = 32632 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36276 req_id = 3eb3663a-24c0-4493-8caf-e0f4b00083be response_code = 200 uri = /crucible/0/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c/snapshots/2f669e1e-7251-4220-ac4e-00bc54e3a4e7
792024-08-27T01:37:00.749ZINFOcrucible-agent (datafile): path is "/data/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c"
802024-08-27T01:37:00.757ZINFOcrucible-agent (datafile): dataset is oxp_c4fd38fc-99ba-4975-a8e6-b8bb5d9becbf/crucible/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
812024-08-27T01:37:00.766ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_c4fd38fc-99ba-4975-a8e6-b8bb5d9becbf/crucible/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c@2f669e1e-7251-4220-ac4e-00bc54e3a4e7
822024-08-27T01:37:00.766ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_c4fd38fc-99ba-4975-a8e6-b8bb5d9becbf/crucible/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c", "2f669e1e-7251-4220-ac4e-00bc54e3a4e7"]
832024-08-27T01:37:00.774ZINFOcrucible-agent (datafile): requesting running snapshot 20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c-2f669e1e-7251-4220-ac4e-00bc54e3a4e7 state: Requested
842024-08-27T01:37:00.774ZINFOcrucible-agent (worker): applying SMF actions for region 20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c running snapshot 2f669e1e-7251-4220-ac4e-00bc54e3a4e7 (state Requested)...
852024-08-27T01:37:00.774ZINFOcrucible-agent (dropshot): request completed latency_us = 25732 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36276 req_id = cbee28dc-865a-4ad4-a9c6-a5a9ed8f51a6 response_code = 200 uri = /crucible/0/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c/snapshots/2f669e1e-7251-4220-ac4e-00bc54e3a4e7/run
862024-08-27T01:37:00.776ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
872024-08-27T01:37:00.776ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
882024-08-27T01:37:00.776ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::16
892024-08-27T01:37:00.776ZINFOcrucible-agent (worker): reconfig required
902024-08-27T01:37:00.776ZINFOcrucible-agent (worker): using existing config property group
912024-08-27T01:37:00.776ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
922024-08-27T01:37:00.777ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
932024-08-27T01:37:00.777ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
942024-08-27T01:37:00.777ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
952024-08-27T01:37:00.777ZINFOcrucible-agent (worker): commit
962024-08-27T01:37:00.784ZINFOcrucible-agent (worker): ok!
972024-08-27T01:37:00.784ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c-2f669e1e-7251-4220-ac4e-00bc54e3a4e7
982024-08-27T01:37:00.791ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c-2f669e1e-7251-4220-ac4e-00bc54e3a4e7
992024-08-27T01:37:00.808ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c-2f669e1e-7251-4220-ac4e-00bc54e3a4e7
1002024-08-27T01:37:00.809ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c/.zfs/snapshot/2f669e1e-7251-4220-ac4e-00bc54e3a4e7
1012024-08-27T01:37:00.809ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1022024-08-27T01:37:00.809ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
1032024-08-27T01:37:00.809ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
1042024-08-27T01:37:00.809ZINFOcrucible-agent (worker): commit
1052024-08-27T01:37:00.813ZINFOcrucible-agent (worker): ok!
1062024-08-27T01:37:00.820ZINFOcrucible-agent (worker): SMF ok!
1072024-08-27T01:37:00.820ZINFOcrucible-agent (datafile): region 20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c running snapshot 2f669e1e-7251-4220-ac4e-00bc54e3a4e7 state: Requested -> Created
1082024-08-27T01:37:00.998ZINFOcrucible-agent (datafile): path is "/data/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c"
1092024-08-27T01:37:01.007ZINFOcrucible-agent (datafile): dataset is oxp_c4fd38fc-99ba-4975-a8e6-b8bb5d9becbf/crucible/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c
1102024-08-27T01:37:01.016ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_c4fd38fc-99ba-4975-a8e6-b8bb5d9becbf/crucible/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c@2f669e1e-7251-4220-ac4e-00bc54e3a4e7
1112024-08-27T01:37:01.016ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_c4fd38fc-99ba-4975-a8e6-b8bb5d9becbf/crucible/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c", "2f669e1e-7251-4220-ac4e-00bc54e3a4e7"]
1122024-08-27T01:37:01.024ZINFOcrucible-agent (dropshot): request completed latency_us = 25698 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36276 req_id = 6653f10c-74c1-40be-a2e0-188e6727a0d6 response_code = 200 uri = /crucible/0/regions/20bd04b0-8ec4-4f9a-ac41-c26bf1ef871c/snapshots/2f669e1e-7251-4220-ac4e-00bc54e3a4e7/run