Max level shown:
1 [ Jul 6 00:07:51 Enabled. ]
2 [ Jul 6 00:07:51 Rereading configuration. ]
3 [ Jul 6 00:07:53 Rereading configuration. ]
4 [ Jul 6 00:07:55 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible -l [fd00:1122:3344:101::14]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-07-06T00:07:55.690ZINFOcrucible-agent: dataset: "oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible"
72024-07-06T00:07:55.695ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::14]:32345
82024-07-06T00:07:55.695ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-07-06T00:07:55.715ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-07-06T00:07:55.827ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::14]:32345
112024-07-06T00:08:59.616ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::c]:35991
122024-07-06T00:08:59.621ZINFOcrucible-agent (datafile): region ca1c0728-f767-41ea-af07-b6f3c894768e state: Requested
132024-07-06T00:08:59.622ZINFOcrucible-agent (dropshot): request completed latency_us = 3660 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:35991 req_id = e2046f3f-029f-412c-aad6-c5b2f3e1ddab response_code = 200 uri = /crucible/0/regions
142024-07-06T00:08:59.622ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152024-07-06T00:08:59.635ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible/regions/ca1c0728-f767-41ea-af07-b6f3c894768e
162024-07-06T00:08:59.635ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible/regions/ca1c0728-f767-41ea-af07-b6f3c894768e
172024-07-06T00:08:59.694ZINFOcrucible-agent (worker): creating region Region { id: RegionId("ca1c0728-f767-41ea-af07-b6f3c894768e"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/ca1c0728-f767-41ea-af07-b6f3c894768e" region = ca1c0728-f767-41ea-af07-b6f3c894768e
182024-07-06T00:08:59.694ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "ca1c0728-f767-41ea-af07-b6f3c894768e" "--data" "/data/regions/ca1c0728-f767-41ea-af07-b6f3c894768e" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = ca1c0728-f767-41ea-af07-b6f3c894768e
192024-07-06T00:08:59.810ZINFOcrucible-agent (worker): region files created ok region = ca1c0728-f767-41ea-af07-b6f3c894768e
202024-07-06T00:08:59.810ZINFOcrucible-agent (datafile): region ca1c0728-f767-41ea-af07-b6f3c894768e state: Requested -> Created
212024-07-06T00:08:59.810ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-07-06T00:08:59.810ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-ca1c0728-f767-41ea-af07-b6f3c894768e
232024-07-06T00:08:59.813ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-ca1c0728-f767-41ea-af07-b6f3c894768e
242024-07-06T00:08:59.822ZINFOcrucible-agent (worker): creating config property group
252024-07-06T00:08:59.825ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ca1c0728-f767-41ea-af07-b6f3c894768e
262024-07-06T00:08:59.826ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ca1c0728-f767-41ea-af07-b6f3c894768e
272024-07-06T00:08:59.826ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-07-06T00:08:59.826ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
292024-07-06T00:08:59.826ZINFOcrucible-agent (worker): commit
302024-07-06T00:08:59.830ZINFOcrucible-agent (worker): ok!
312024-07-06T00:08:59.835ZINFOcrucible-agent (worker): SMF ok!
322024-07-06T00:08:59.931ZINFOcrucible-agent (dropshot): request completed latency_us = 89 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:35991 req_id = c51cb216-2cf4-41d0-9aeb-00df73019b90 response_code = 200 uri = /crucible/0/regions
332024-07-06T00:09:00.446ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:35991 req_id = 1cd44fa1-c3b6-4690-97b1-ba4cbae81b45 response_code = 200 uri = /crucible/0/regions/ca1c0728-f767-41ea-af07-b6f3c894768e
342024-07-06T00:09:00.448ZINFOcrucible-agent (datafile): path is "/data/regions/ca1c0728-f767-41ea-af07-b6f3c894768e"
352024-07-06T00:09:00.457ZINFOcrucible-agent (datafile): dataset is oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible/regions/ca1c0728-f767-41ea-af07-b6f3c894768e
362024-07-06T00:09:00.465ZINFOcrucible-agent (datafile): region ca1c0728-f767-41ea-af07-b6f3c894768e state: Created -> Tombstoned
372024-07-06T00:09:00.465ZINFOcrucible-agent (worker): applying SMF actions before removal...
382024-07-06T00:09:00.465ZINFOcrucible-agent (dropshot): request completed latency_us = 16939 local_addr = [fd00:1122:3344:101::14]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:35991 req_id = 2380d0db-fea9-4fbb-86b2-9d514361b7d0 response_code = 204 uri = /crucible/0/regions/ca1c0728-f767-41ea-af07-b6f3c894768e
392024-07-06T00:09:00.466ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-ca1c0728-f767-41ea-af07-b6f3c894768e (instance states: (Some(Online), None))
402024-07-06T00:09:00.468ZINFOcrucible-agent (dropshot): request completed latency_us = 25 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:35991 req_id = 9048038d-2647-4e35-9b10-f55eeb471b84 response_code = 200 uri = /crucible/0/regions/ca1c0728-f767-41ea-af07-b6f3c894768e
412024-07-06T00:09:00.471ZINFOcrucible-agent (worker): SMF ok!
422024-07-06T00:09:00.481ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible/regions/ca1c0728-f767-41ea-af07-b6f3c894768e" region = ca1c0728-f767-41ea-af07-b6f3c894768e
432024-07-06T00:09:00.535ZINFOcrucible-agent (datafile): region ca1c0728-f767-41ea-af07-b6f3c894768e state: Tombstoned -> Destroyed
442024-07-06T00:09:00.578ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:35991 req_id = 5069c2ce-e662-4ed7-aab2-c800514cc3bc response_code = 200 uri = /crucible/0/regions/ca1c0728-f767-41ea-af07-b6f3c894768e
452024-07-06T00:09:08.173ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:33129
462024-07-06T00:09:08.173ZINFOcrucible-agent (datafile): region 866e802e-9fe2-4d11-aadf-37b953c9c772 state: Requested
472024-07-06T00:09:08.173ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
482024-07-06T00:09:08.173ZINFOcrucible-agent (dropshot): request completed latency_us = 292 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33129 req_id = 94146b95-e3cf-4747-8bf2-2fb44a8a9523 response_code = 200 uri = /crucible/0/regions
492024-07-06T00:09:08.183ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible/regions/866e802e-9fe2-4d11-aadf-37b953c9c772
502024-07-06T00:09:08.183ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible/regions/866e802e-9fe2-4d11-aadf-37b953c9c772
512024-07-06T00:09:08.247ZINFOcrucible-agent (worker): creating region Region { id: RegionId("866e802e-9fe2-4d11-aadf-37b953c9c772"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/866e802e-9fe2-4d11-aadf-37b953c9c772" region = 866e802e-9fe2-4d11-aadf-37b953c9c772
522024-07-06T00:09:08.250ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "866e802e-9fe2-4d11-aadf-37b953c9c772" "--data" "/data/regions/866e802e-9fe2-4d11-aadf-37b953c9c772" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 866e802e-9fe2-4d11-aadf-37b953c9c772
532024-07-06T00:09:08.313ZINFOcrucible-agent (worker): region files created ok region = 866e802e-9fe2-4d11-aadf-37b953c9c772
542024-07-06T00:09:08.313ZINFOcrucible-agent (datafile): region 866e802e-9fe2-4d11-aadf-37b953c9c772 state: Requested -> Created
552024-07-06T00:09:08.313ZINFOcrucible-agent (worker): applying SMF actions post create...
562024-07-06T00:09:08.315ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-866e802e-9fe2-4d11-aadf-37b953c9c772
572024-07-06T00:09:08.318ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-866e802e-9fe2-4d11-aadf-37b953c9c772
582024-07-06T00:09:08.326ZINFOcrucible-agent (worker): creating config property group
592024-07-06T00:09:08.330ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-866e802e-9fe2-4d11-aadf-37b953c9c772
602024-07-06T00:09:08.330ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/866e802e-9fe2-4d11-aadf-37b953c9c772
612024-07-06T00:09:08.330ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
622024-07-06T00:09:08.330ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
632024-07-06T00:09:08.330ZINFOcrucible-agent (worker): commit
642024-07-06T00:09:08.333ZINFOcrucible-agent (worker): ok!
652024-07-06T00:09:08.336ZINFOcrucible-agent (dropshot): request completed latency_us = 63 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33129 req_id = 2fb2fedf-7dbc-4a53-9313-b22ae1c3f5d5 response_code = 200 uri = /crucible/0/regions
662024-07-06T00:09:08.338ZINFOcrucible-agent (worker): SMF ok!
672024-07-06T00:14:29.582ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:56130
682024-07-06T00:14:29.582ZINFOcrucible-agent (datafile): region 0b789859-1ce3-4b9e-8172-a0b2a1024dd9 state: Requested
692024-07-06T00:14:29.582ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
702024-07-06T00:14:29.582ZINFOcrucible-agent (dropshot): request completed latency_us = 622 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56130 req_id = d5082784-ea0b-4d13-8d72-2d0fef59b008 response_code = 200 uri = /crucible/0/regions
712024-07-06T00:14:29.593ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible/regions/0b789859-1ce3-4b9e-8172-a0b2a1024dd9
722024-07-06T00:14:29.593ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible/regions/0b789859-1ce3-4b9e-8172-a0b2a1024dd9
732024-07-06T00:14:29.655ZINFOcrucible-agent (worker): creating region Region { id: RegionId("0b789859-1ce3-4b9e-8172-a0b2a1024dd9"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/0b789859-1ce3-4b9e-8172-a0b2a1024dd9" region = 0b789859-1ce3-4b9e-8172-a0b2a1024dd9
742024-07-06T00:14:29.655ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "0b789859-1ce3-4b9e-8172-a0b2a1024dd9" "--data" "/data/regions/0b789859-1ce3-4b9e-8172-a0b2a1024dd9" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 0b789859-1ce3-4b9e-8172-a0b2a1024dd9
752024-07-06T00:14:29.726ZINFOcrucible-agent (worker): region files created ok region = 0b789859-1ce3-4b9e-8172-a0b2a1024dd9
762024-07-06T00:14:29.726ZINFOcrucible-agent (datafile): region 0b789859-1ce3-4b9e-8172-a0b2a1024dd9 state: Requested -> Created
772024-07-06T00:14:29.726ZINFOcrucible-agent (worker): applying SMF actions post create...
782024-07-06T00:14:29.727ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-0b789859-1ce3-4b9e-8172-a0b2a1024dd9
792024-07-06T00:14:29.736ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-0b789859-1ce3-4b9e-8172-a0b2a1024dd9
802024-07-06T00:14:29.751ZINFOcrucible-agent (worker): creating config property group
812024-07-06T00:14:29.753ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0b789859-1ce3-4b9e-8172-a0b2a1024dd9
822024-07-06T00:14:29.754ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0b789859-1ce3-4b9e-8172-a0b2a1024dd9
832024-07-06T00:14:29.754ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
842024-07-06T00:14:29.754ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
852024-07-06T00:14:29.754ZINFOcrucible-agent (worker): commit
862024-07-06T00:14:29.757ZINFOcrucible-agent (worker): ok!
872024-07-06T00:14:29.762ZINFOcrucible-agent (worker): SMF ok!
882024-07-06T00:14:29.909ZINFOcrucible-agent (dropshot): request completed latency_us = 112 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56130 req_id = 813f0f06-fcf2-4ca0-9d53-f6e3b8b6494a response_code = 200 uri = /crucible/0/regions
892024-07-06T00:14:30.948ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:47541
902024-07-06T00:14:30.948ZINFOcrucible-agent (dropshot): request completed latency_us = 45 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:47541 req_id = 0efefdf4-9f8b-423f-adf2-1323de1841eb response_code = 200 uri = /crucible/0/regions/866e802e-9fe2-4d11-aadf-37b953c9c772
912024-07-06T00:14:30.948ZINFOcrucible-agent (datafile): path is "/data/regions/866e802e-9fe2-4d11-aadf-37b953c9c772"
922024-07-06T00:14:30.957ZINFOcrucible-agent (datafile): dataset is oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible/regions/866e802e-9fe2-4d11-aadf-37b953c9c772
932024-07-06T00:14:30.967ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible/regions/866e802e-9fe2-4d11-aadf-37b953c9c772@3916d487-df2b-42eb-b478-d4b46764e516
942024-07-06T00:14:30.967ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible/regions/866e802e-9fe2-4d11-aadf-37b953c9c772", "3916d487-df2b-42eb-b478-d4b46764e516"]
952024-07-06T00:14:30.978ZINFOcrucible-agent (dropshot): request completed latency_us = 29693 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:47541 req_id = 4096cd24-2692-464b-93e8-93d5d6f96dcd response_code = 200 uri = /crucible/0/regions/866e802e-9fe2-4d11-aadf-37b953c9c772/snapshots/3916d487-df2b-42eb-b478-d4b46764e516
962024-07-06T00:14:30.978ZINFOcrucible-agent (datafile): path is "/data/regions/866e802e-9fe2-4d11-aadf-37b953c9c772"
972024-07-06T00:14:30.987ZINFOcrucible-agent (datafile): dataset is oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible/regions/866e802e-9fe2-4d11-aadf-37b953c9c772
982024-07-06T00:14:30.995ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible/regions/866e802e-9fe2-4d11-aadf-37b953c9c772@3916d487-df2b-42eb-b478-d4b46764e516
992024-07-06T00:14:30.996ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_187c3b21-21e5-4726-9974-0290b3e4dbd4/crucible/regions/866e802e-9fe2-4d11-aadf-37b953c9c772", "3916d487-df2b-42eb-b478-d4b46764e516"]
1002024-07-06T00:14:31.008ZINFOcrucible-agent (datafile): requesting running snapshot 866e802e-9fe2-4d11-aadf-37b953c9c772-3916d487-df2b-42eb-b478-d4b46764e516 state: Requested
1012024-07-06T00:14:31.008ZINFOcrucible-agent (worker): applying SMF actions for region 866e802e-9fe2-4d11-aadf-37b953c9c772 running snapshot 3916d487-df2b-42eb-b478-d4b46764e516 (state Requested)...
1022024-07-06T00:14:31.008ZINFOcrucible-agent (dropshot): request completed latency_us = 29566 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:47541 req_id = e1625e7f-6d65-4985-98d8-48bbc5b28870 response_code = 200 uri = /crucible/0/regions/866e802e-9fe2-4d11-aadf-37b953c9c772/snapshots/3916d487-df2b-42eb-b478-d4b46764e516/run
1032024-07-06T00:14:31.011ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-866e802e-9fe2-4d11-aadf-37b953c9c772-3916d487-df2b-42eb-b478-d4b46764e516
1042024-07-06T00:14:31.014ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-866e802e-9fe2-4d11-aadf-37b953c9c772-3916d487-df2b-42eb-b478-d4b46764e516
1052024-07-06T00:14:31.023ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-866e802e-9fe2-4d11-aadf-37b953c9c772-3916d487-df2b-42eb-b478-d4b46764e516
1062024-07-06T00:14:31.023ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/866e802e-9fe2-4d11-aadf-37b953c9c772/.zfs/snapshot/3916d487-df2b-42eb-b478-d4b46764e516
1072024-07-06T00:14:31.023ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1082024-07-06T00:14:31.023ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
1092024-07-06T00:14:31.023ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
1102024-07-06T00:14:31.023ZINFOcrucible-agent (worker): commit
1112024-07-06T00:14:31.027ZINFOcrucible-agent (worker): ok!
1122024-07-06T00:14:31.031ZINFOcrucible-agent (worker): SMF ok!
1132024-07-06T00:14:31.031ZINFOcrucible-agent (datafile): region 866e802e-9fe2-4d11-aadf-37b953c9c772 running snapshot 3916d487-df2b-42eb-b478-d4b46764e516 state: Requested -> Created