Max level shown:
1 [ Apr 26 00:20:11 Enabled. ]
2 [ Apr 26 00:20:11 Rereading configuration. ]
3 [ Apr 26 00:20:13 Rereading configuration. ]
4 [ Apr 26 00:20:15 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_d264dd09-dd1e-4b44-ac20-70d4e8728824/crucible -l [fd00:1122:3344:101::12]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-04-26T00:20:15.846ZINFOcrucible-agent: dataset: "oxp_d264dd09-dd1e-4b44-ac20-70d4e8728824/crucible"
72024-04-26T00:20:15.848ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::12]:32345
82024-04-26T00:20:15.848ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-04-26T00:20:15.866ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-04-26T00:20:15.963ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::12]:32345
112024-04-26T00:20:59.409ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::c]:64533
122024-04-26T00:20:59.411ZINFOcrucible-agent (datafile): region e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c state: Requested
132024-04-26T00:20:59.412ZINFOcrucible-agent (dropshot): request completed latency_us = 1556 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:64533 req_id = 2d18c00b-ae59-47a5-bba3-5e147d374df3 response_code = 200 uri = /crucible/0/regions
142024-04-26T00:20:59.412ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152024-04-26T00:20:59.424ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_d264dd09-dd1e-4b44-ac20-70d4e8728824/crucible/regions/e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
162024-04-26T00:20:59.424ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_d264dd09-dd1e-4b44-ac20-70d4e8728824/crucible/regions/e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
172024-04-26T00:20:59.486ZINFOcrucible-agent (worker): creating region Region { id: RegionId("e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c" region = e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
182024-04-26T00:20:59.488ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c" "--data" "/data/regions/e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
192024-04-26T00:20:59.565ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:64533 req_id = 4aa7d46d-7e3f-4d83-8ff3-e4f900cd2fcf response_code = 200 uri = /crucible/0/regions
202024-04-26T00:20:59.598ZINFOcrucible-agent (worker): region files created ok region = e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
212024-04-26T00:20:59.598ZINFOcrucible-agent (datafile): region e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c state: Requested -> Created
222024-04-26T00:20:59.598ZINFOcrucible-agent (worker): applying SMF actions post create...
232024-04-26T00:20:59.598ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
242024-04-26T00:20:59.602ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
252024-04-26T00:20:59.612ZINFOcrucible-agent (worker): creating config property group
262024-04-26T00:20:59.615ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
272024-04-26T00:20:59.615ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
282024-04-26T00:20:59.618ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-04-26T00:20:59.618ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
302024-04-26T00:20:59.618ZINFOcrucible-agent (worker): commit
312024-04-26T00:20:59.625ZINFOcrucible-agent (worker): ok!
322024-04-26T00:20:59.629ZINFOcrucible-agent (worker): SMF ok!
332024-04-26T00:20:59.842ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:64533 req_id = aa40b9be-859c-41fa-a584-428c6092f377 response_code = 200 uri = /crucible/0/regions
342024-04-26T00:21:00.500ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::c]:54830
352024-04-26T00:21:00.500ZINFOcrucible-agent (dropshot): request completed latency_us = 66 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:54830 req_id = 7697d2a7-d951-419d-a393-94a4c19aac2e response_code = 200 uri = /crucible/0/regions/e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
362024-04-26T00:21:00.500ZINFOcrucible-agent (datafile): path is "/data/regions/e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c"
372024-04-26T00:21:00.511ZINFOcrucible-agent (datafile): dataset is oxp_d264dd09-dd1e-4b44-ac20-70d4e8728824/crucible/regions/e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
382024-04-26T00:21:00.519ZINFOcrucible-agent (datafile): region e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c state: Created -> Tombstoned
392024-04-26T00:21:00.519ZINFOcrucible-agent (worker): applying SMF actions before removal...
402024-04-26T00:21:00.519ZINFOcrucible-agent (dropshot): request completed latency_us = 19459 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:54830 req_id = 1475f12d-5f53-4831-be7b-849819f6b89a response_code = 204 uri = /crucible/0/regions/e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
412024-04-26T00:21:00.520ZINFOcrucible-agent (dropshot): request completed latency_us = 25 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:54830 req_id = 6ecfb967-4117-4a37-a48c-1e3f9f9dbda1 response_code = 200 uri = /crucible/0/regions/e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
422024-04-26T00:21:00.521ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c (instance states: (Some(Online), None))
432024-04-26T00:21:00.525ZINFOcrucible-agent (worker): SMF ok!
442024-04-26T00:21:00.537ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_d264dd09-dd1e-4b44-ac20-70d4e8728824/crucible/regions/e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c" region = e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
452024-04-26T00:21:00.592ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:54830 req_id = 405c6e94-71f2-4000-9a3f-512849cb4849 response_code = 200 uri = /crucible/0/regions/e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
462024-04-26T00:21:00.593ZINFOcrucible-agent (datafile): region e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c state: Tombstoned -> Destroyed
472024-04-26T00:21:00.712ZINFOcrucible-agent (dropshot): request completed latency_us = 31 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:54830 req_id = edd2bd10-a4fb-47b9-8651-0620a7938af0 response_code = 200 uri = /crucible/0/regions/e1b2e74d-8fa9-432e-b95d-6fd854f6cf1c
482024-04-26T00:26:55.276ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:40527
492024-04-26T00:26:55.403ZINFOcrucible-agent (datafile): region f2cd00e7-8d1a-483c-870b-8f5ae88bd72b state: Requested
502024-04-26T00:26:55.403ZINFOcrucible-agent (dropshot): request completed latency_us = 340 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40527 req_id = 3fb1e7e1-9a03-4099-9408-ed52c525fb43 response_code = 200 uri = /crucible/0/regions
512024-04-26T00:26:55.403ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
522024-04-26T00:26:55.413ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_d264dd09-dd1e-4b44-ac20-70d4e8728824/crucible/regions/f2cd00e7-8d1a-483c-870b-8f5ae88bd72b
532024-04-26T00:26:55.413ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_d264dd09-dd1e-4b44-ac20-70d4e8728824/crucible/regions/f2cd00e7-8d1a-483c-870b-8f5ae88bd72b
542024-04-26T00:26:55.476ZINFOcrucible-agent (worker): creating region Region { id: RegionId("f2cd00e7-8d1a-483c-870b-8f5ae88bd72b"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/f2cd00e7-8d1a-483c-870b-8f5ae88bd72b" region = f2cd00e7-8d1a-483c-870b-8f5ae88bd72b
552024-04-26T00:26:55.478ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "f2cd00e7-8d1a-483c-870b-8f5ae88bd72b" "--data" "/data/regions/f2cd00e7-8d1a-483c-870b-8f5ae88bd72b" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = f2cd00e7-8d1a-483c-870b-8f5ae88bd72b
562024-04-26T00:26:55.574ZINFOcrucible-agent (dropshot): request completed latency_us = 746 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40527 req_id = 648f9231-40f3-416f-b58c-fdadf623cadf response_code = 200 uri = /crucible/0/regions
572024-04-26T00:26:55.625ZINFOcrucible-agent (worker): region files created ok region = f2cd00e7-8d1a-483c-870b-8f5ae88bd72b
582024-04-26T00:26:55.625ZINFOcrucible-agent (datafile): region f2cd00e7-8d1a-483c-870b-8f5ae88bd72b state: Requested -> Created
592024-04-26T00:26:55.625ZINFOcrucible-agent (worker): applying SMF actions post create...
602024-04-26T00:26:55.626ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-f2cd00e7-8d1a-483c-870b-8f5ae88bd72b
612024-04-26T00:26:55.631ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-f2cd00e7-8d1a-483c-870b-8f5ae88bd72b
622024-04-26T00:26:55.646ZINFOcrucible-agent (worker): creating config property group
632024-04-26T00:26:55.649ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-f2cd00e7-8d1a-483c-870b-8f5ae88bd72b
642024-04-26T00:26:55.650ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/f2cd00e7-8d1a-483c-870b-8f5ae88bd72b
652024-04-26T00:26:55.650ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
662024-04-26T00:26:55.650ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
672024-04-26T00:26:55.650ZINFOcrucible-agent (worker): commit
682024-04-26T00:26:55.653ZINFOcrucible-agent (worker): ok!
692024-04-26T00:26:55.657ZINFOcrucible-agent (worker): SMF ok!
702024-04-26T00:26:55.878ZINFOcrucible-agent (dropshot): request completed latency_us = 132 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40527 req_id = a74dfbd8-aa1a-4f59-b141-174242ffae4c response_code = 200 uri = /crucible/0/regions