Max level shown:
1 [ Jul 22 23:07:15 Enabled. ]
2 [ Jul 22 23:07:15 Rereading configuration. ]
3 [ Jul 22 23:07:18 Rereading configuration. ]
4 [ Jul 22 23:07:22 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_166fc5d5-ed26-44e3-acae-034d165b26da/crucible -l [fd00:1122:3344:101::15]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-07-22T23:07:22.798ZINFOcrucible-agent: dataset: "oxp_166fc5d5-ed26-44e3-acae-034d165b26da/crucible"
72024-07-22T23:07:22.801ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::15]:32345
82024-07-22T23:07:22.801ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-07-22T23:07:22.816ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-07-22T23:07:22.918ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::15]:32345
112024-07-22T23:08:34.401ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:46894
122024-07-22T23:08:34.420ZINFOcrucible-agent (datafile): region 511c670d-afe5-4745-86de-0cbbc680ae3b state: Requested
132024-07-22T23:08:34.427ZINFOcrucible-agent (dropshot): request completed latency_us = 14650 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46894 req_id = e9f637b4-3aa0-4899-be88-401f4eda6451 response_code = 200 uri = /crucible/0/regions
142024-07-22T23:08:34.427ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152024-07-22T23:08:34.440ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_166fc5d5-ed26-44e3-acae-034d165b26da/crucible/regions/511c670d-afe5-4745-86de-0cbbc680ae3b
162024-07-22T23:08:34.440ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_166fc5d5-ed26-44e3-acae-034d165b26da/crucible/regions/511c670d-afe5-4745-86de-0cbbc680ae3b
172024-07-22T23:08:34.496ZINFOcrucible-agent (worker): creating region Region { id: RegionId("511c670d-afe5-4745-86de-0cbbc680ae3b"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/511c670d-afe5-4745-86de-0cbbc680ae3b" region = 511c670d-afe5-4745-86de-0cbbc680ae3b
182024-07-22T23:08:34.499ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "511c670d-afe5-4745-86de-0cbbc680ae3b" "--data" "/data/regions/511c670d-afe5-4745-86de-0cbbc680ae3b" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 511c670d-afe5-4745-86de-0cbbc680ae3b
192024-07-22T23:08:34.654ZINFOcrucible-agent (worker): region files created ok region = 511c670d-afe5-4745-86de-0cbbc680ae3b
202024-07-22T23:08:34.654ZINFOcrucible-agent (datafile): region 511c670d-afe5-4745-86de-0cbbc680ae3b state: Requested -> Created
212024-07-22T23:08:34.654ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-07-22T23:08:34.654ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-511c670d-afe5-4745-86de-0cbbc680ae3b
232024-07-22T23:08:34.661ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-511c670d-afe5-4745-86de-0cbbc680ae3b
242024-07-22T23:08:34.678ZINFOcrucible-agent (worker): creating config property group
252024-07-22T23:08:34.681ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-511c670d-afe5-4745-86de-0cbbc680ae3b
262024-07-22T23:08:34.682ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/511c670d-afe5-4745-86de-0cbbc680ae3b
272024-07-22T23:08:34.686ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-07-22T23:08:34.686ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
292024-07-22T23:08:34.686ZINFOcrucible-agent (worker): commit
302024-07-22T23:08:34.692ZINFOcrucible-agent (worker): ok!
312024-07-22T23:08:34.699ZINFOcrucible-agent (worker): SMF ok!
322024-07-22T23:08:34.761ZINFOcrucible-agent (dropshot): request completed latency_us = 77 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46894 req_id = 888fa749-7935-4ac2-8942-6ee2915b7428 response_code = 200 uri = /crucible/0/regions
332024-07-22T23:13:42.702ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:46552
342024-07-22T23:13:42.705ZINFOcrucible-agent (dropshot): request completed latency_us = 3097 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:46552 req_id = 41a46713-dd17-49de-81fd-03639878dd01 response_code = 200 uri = /crucible/0/regions/511c670d-afe5-4745-86de-0cbbc680ae3b
352024-07-22T23:13:42.707ZINFOcrucible-agent (datafile): path is "/data/regions/511c670d-afe5-4745-86de-0cbbc680ae3b"
362024-07-22T23:13:42.716ZINFOcrucible-agent (datafile): dataset is oxp_166fc5d5-ed26-44e3-acae-034d165b26da/crucible/regions/511c670d-afe5-4745-86de-0cbbc680ae3b
372024-07-22T23:13:42.724ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_166fc5d5-ed26-44e3-acae-034d165b26da/crucible/regions/511c670d-afe5-4745-86de-0cbbc680ae3b@9a8a0026-eb67-4908-babd-b4c60882cf96
382024-07-22T23:13:42.724ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_166fc5d5-ed26-44e3-acae-034d165b26da/crucible/regions/511c670d-afe5-4745-86de-0cbbc680ae3b", "9a8a0026-eb67-4908-babd-b4c60882cf96"]
392024-07-22T23:13:42.736ZINFOcrucible-agent (dropshot): request completed latency_us = 31369 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:46552 req_id = d8087a75-7ade-484d-873a-b3afb75a6bf6 response_code = 200 uri = /crucible/0/regions/511c670d-afe5-4745-86de-0cbbc680ae3b/snapshots/9a8a0026-eb67-4908-babd-b4c60882cf96
402024-07-22T23:13:42.737ZINFOcrucible-agent (datafile): path is "/data/regions/511c670d-afe5-4745-86de-0cbbc680ae3b"
412024-07-22T23:13:42.744ZINFOcrucible-agent (datafile): dataset is oxp_166fc5d5-ed26-44e3-acae-034d165b26da/crucible/regions/511c670d-afe5-4745-86de-0cbbc680ae3b
422024-07-22T23:13:42.752ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_166fc5d5-ed26-44e3-acae-034d165b26da/crucible/regions/511c670d-afe5-4745-86de-0cbbc680ae3b@9a8a0026-eb67-4908-babd-b4c60882cf96
432024-07-22T23:13:42.752ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_166fc5d5-ed26-44e3-acae-034d165b26da/crucible/regions/511c670d-afe5-4745-86de-0cbbc680ae3b", "9a8a0026-eb67-4908-babd-b4c60882cf96"]
442024-07-22T23:13:42.762ZINFOcrucible-agent (datafile): requesting running snapshot 511c670d-afe5-4745-86de-0cbbc680ae3b-9a8a0026-eb67-4908-babd-b4c60882cf96 state: Requested
452024-07-22T23:13:42.763ZINFOcrucible-agent (worker): applying SMF actions for region 511c670d-afe5-4745-86de-0cbbc680ae3b running snapshot 9a8a0026-eb67-4908-babd-b4c60882cf96 (state Requested)...
462024-07-22T23:13:42.763ZINFOcrucible-agent (dropshot): request completed latency_us = 26037 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46552 req_id = 11d113ed-3464-4ea8-b2c4-262c568259c1 response_code = 200 uri = /crucible/0/regions/511c670d-afe5-4745-86de-0cbbc680ae3b/snapshots/9a8a0026-eb67-4908-babd-b4c60882cf96/run
472024-07-22T23:13:42.771ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/511c670d-afe5-4745-86de-0cbbc680ae3b
482024-07-22T23:13:42.771ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-07-22T23:13:42.771ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::15
502024-07-22T23:13:42.771ZINFOcrucible-agent (worker): reconfig required
512024-07-22T23:13:42.771ZINFOcrucible-agent (worker): using existing config property group
522024-07-22T23:13:42.771ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-511c670d-afe5-4745-86de-0cbbc680ae3b
532024-07-22T23:13:42.771ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/511c670d-afe5-4745-86de-0cbbc680ae3b
542024-07-22T23:13:42.771ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-07-22T23:13:42.771ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
562024-07-22T23:13:42.771ZINFOcrucible-agent (worker): commit
572024-07-22T23:13:42.787ZINFOcrucible-agent (worker): ok!
582024-07-22T23:13:42.788ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-511c670d-afe5-4745-86de-0cbbc680ae3b-9a8a0026-eb67-4908-babd-b4c60882cf96
592024-07-22T23:13:42.798ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-511c670d-afe5-4745-86de-0cbbc680ae3b-9a8a0026-eb67-4908-babd-b4c60882cf96
602024-07-22T23:13:42.821ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-511c670d-afe5-4745-86de-0cbbc680ae3b-9a8a0026-eb67-4908-babd-b4c60882cf96
612024-07-22T23:13:42.822ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/511c670d-afe5-4745-86de-0cbbc680ae3b/.zfs/snapshot/9a8a0026-eb67-4908-babd-b4c60882cf96
622024-07-22T23:13:42.822ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
632024-07-22T23:13:42.822ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-07-22T23:13:42.822ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
652024-07-22T23:13:42.822ZINFOcrucible-agent (worker): commit
662024-07-22T23:13:42.829ZINFOcrucible-agent (worker): ok!
672024-07-22T23:13:42.836ZINFOcrucible-agent (worker): SMF ok!
682024-07-22T23:13:42.836ZINFOcrucible-agent (datafile): region 511c670d-afe5-4745-86de-0cbbc680ae3b running snapshot 9a8a0026-eb67-4908-babd-b4c60882cf96 state: Requested -> Created
692024-07-22T23:13:45.934ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:37295
702024-07-22T23:13:45.934ZINFOcrucible-agent (datafile): region cdacc612-608e-4145-8519-d8a0eed1faa6 state: Requested
712024-07-22T23:13:45.937ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
722024-07-22T23:13:45.937ZINFOcrucible-agent (dropshot): request completed latency_us = 2237 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:37295 req_id = 6543c7a9-a4d8-42c1-a783-e50c315771d4 response_code = 200 uri = /crucible/0/regions
732024-07-22T23:13:45.947ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_166fc5d5-ed26-44e3-acae-034d165b26da/crucible/regions/cdacc612-608e-4145-8519-d8a0eed1faa6
742024-07-22T23:13:45.947ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_166fc5d5-ed26-44e3-acae-034d165b26da/crucible/regions/cdacc612-608e-4145-8519-d8a0eed1faa6
752024-07-22T23:13:46.044ZINFOcrucible-agent (worker): creating region Region { id: RegionId("cdacc612-608e-4145-8519-d8a0eed1faa6"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/cdacc612-608e-4145-8519-d8a0eed1faa6" region = cdacc612-608e-4145-8519-d8a0eed1faa6
762024-07-22T23:13:46.044ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "cdacc612-608e-4145-8519-d8a0eed1faa6" "--data" "/data/regions/cdacc612-608e-4145-8519-d8a0eed1faa6" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = cdacc612-608e-4145-8519-d8a0eed1faa6
772024-07-22T23:13:46.147ZINFOcrucible-agent (dropshot): request completed latency_us = 93 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:37295 req_id = 7fb856ad-43c9-4a85-89ef-6427bf235b8c response_code = 200 uri = /crucible/0/regions
782024-07-22T23:13:46.223ZINFOcrucible-agent (worker): region files created ok region = cdacc612-608e-4145-8519-d8a0eed1faa6
792024-07-22T23:13:46.223ZINFOcrucible-agent (datafile): region cdacc612-608e-4145-8519-d8a0eed1faa6 state: Requested -> Created
802024-07-22T23:13:46.223ZINFOcrucible-agent (worker): applying SMF actions post create...
812024-07-22T23:13:46.224ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/511c670d-afe5-4745-86de-0cbbc680ae3b
822024-07-22T23:13:46.224ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
832024-07-22T23:13:46.224ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::15
842024-07-22T23:13:46.224ZINFOcrucible-agent (worker): reconfig required
852024-07-22T23:13:46.224ZINFOcrucible-agent (worker): using existing config property group
862024-07-22T23:13:46.224ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-511c670d-afe5-4745-86de-0cbbc680ae3b
872024-07-22T23:13:46.224ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/511c670d-afe5-4745-86de-0cbbc680ae3b
882024-07-22T23:13:46.224ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
892024-07-22T23:13:46.224ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
902024-07-22T23:13:46.224ZINFOcrucible-agent (worker): commit
912024-07-22T23:13:46.233ZINFOcrucible-agent (worker): ok!
922024-07-22T23:13:46.234ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-cdacc612-608e-4145-8519-d8a0eed1faa6
932024-07-22T23:13:46.241ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-cdacc612-608e-4145-8519-d8a0eed1faa6
942024-07-22T23:13:46.264ZINFOcrucible-agent (worker): creating config property group
952024-07-22T23:13:46.273ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-cdacc612-608e-4145-8519-d8a0eed1faa6
962024-07-22T23:13:46.274ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/cdacc612-608e-4145-8519-d8a0eed1faa6
972024-07-22T23:13:46.274ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
982024-07-22T23:13:46.274ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
992024-07-22T23:13:46.274ZINFOcrucible-agent (worker): commit
1002024-07-22T23:13:46.281ZINFOcrucible-agent (worker): ok!
1012024-07-22T23:13:46.291ZINFOcrucible-agent (worker): SMF ok!
1022024-07-22T23:13:46.831ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:37295 req_id = 49779198-1691-4a73-b370-7a7bf0f682ec response_code = 200 uri = /crucible/0/regions
1032024-07-22T23:14:10.992ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:37295 req_id = 14dabd1e-3922-44d2-b748-31939710d001 response_code = 200 uri = /crucible/0/regions/cdacc612-608e-4145-8519-d8a0eed1faa6
1042024-07-22T23:14:10.995ZINFOcrucible-agent (datafile): path is "/data/regions/cdacc612-608e-4145-8519-d8a0eed1faa6"
1052024-07-22T23:14:11.008ZINFOcrucible-agent (datafile): dataset is oxp_166fc5d5-ed26-44e3-acae-034d165b26da/crucible/regions/cdacc612-608e-4145-8519-d8a0eed1faa6
1062024-07-22T23:14:11.017ZINFOcrucible-agent (datafile): region cdacc612-608e-4145-8519-d8a0eed1faa6 state: Created -> Tombstoned
1072024-07-22T23:14:11.018ZINFOcrucible-agent (worker): applying SMF actions before removal...
1082024-07-22T23:14:11.018ZINFOcrucible-agent (dropshot): request completed latency_us = 23724 local_addr = [fd00:1122:3344:101::15]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:37295 req_id = c51645b8-8462-40be-af92-db767ceca41c response_code = 204 uri = /crucible/0/regions/cdacc612-608e-4145-8519-d8a0eed1faa6
1092024-07-22T23:14:11.019ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-cdacc612-608e-4145-8519-d8a0eed1faa6 (instance states: (Some(Online), None))
1102024-07-22T23:14:11.020ZINFOcrucible-agent (dropshot): request completed latency_us = 32 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:37295 req_id = 1310c542-dcf6-4d82-86ec-ccb87e93af5c response_code = 200 uri = /crucible/0/regions/cdacc612-608e-4145-8519-d8a0eed1faa6
1112024-07-22T23:14:11.024ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/511c670d-afe5-4745-86de-0cbbc680ae3b
1122024-07-22T23:14:11.025ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1132024-07-22T23:14:11.025ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::15
1142024-07-22T23:14:11.025ZINFOcrucible-agent (worker): reconfig required
1152024-07-22T23:14:11.025ZINFOcrucible-agent (worker): using existing config property group
1162024-07-22T23:14:11.025ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-511c670d-afe5-4745-86de-0cbbc680ae3b
1172024-07-22T23:14:11.025ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/511c670d-afe5-4745-86de-0cbbc680ae3b
1182024-07-22T23:14:11.025ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1192024-07-22T23:14:11.025ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
1202024-07-22T23:14:11.025ZINFOcrucible-agent (worker): commit
1212024-07-22T23:14:11.030ZINFOcrucible-agent (worker): ok!
1222024-07-22T23:14:11.032ZINFOcrucible-agent (worker): SMF ok!
1232024-07-22T23:14:11.041ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_166fc5d5-ed26-44e3-acae-034d165b26da/crucible/regions/cdacc612-608e-4145-8519-d8a0eed1faa6" region = cdacc612-608e-4145-8519-d8a0eed1faa6
1242024-07-22T23:14:11.089ZINFOcrucible-agent (datafile): region cdacc612-608e-4145-8519-d8a0eed1faa6 state: Tombstoned -> Destroyed
1252024-07-22T23:14:11.137ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:37295 req_id = e10f503f-3f8e-4803-bb8d-45e6b4ca57a9 response_code = 200 uri = /crucible/0/regions/cdacc612-608e-4145-8519-d8a0eed1faa6