Max level shown:
1 [ Sep 3 23:51:12 Enabled. ]
2 [ Sep 3 23:51:12 Rereading configuration. ]
3 [ Sep 3 23:51:13 Rereading configuration. ]
4 [ Sep 3 23:51:14 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_96307640-3db4-4a6e-b03a-71a80f6a607d/crucible -l [fd00:1122:3344:101::18]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-09-03T23:51:14.930ZINFOcrucible-agent: dataset: "oxp_96307640-3db4-4a6e-b03a-71a80f6a607d/crucible"
72024-09-03T23:51:14.935ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::18]:32345
82024-09-03T23:51:14.935ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-09-03T23:51:14.962ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-09-03T23:51:15.078ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::18]:32345
112024-09-03T23:56:55.924ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:37855
122024-09-03T23:56:55.959ZINFOcrucible-agent (datafile): region 001e5a02-2834-48f1-8d7f-21783d57710a state: Requested
132024-09-03T23:56:55.963ZINFOcrucible-agent (dropshot): request completed latency_us = 20105 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:37855 req_id = 1db463b7-0cfe-40ab-99f9-11bfd6507fe2 response_code = 200 uri = /crucible/0/regions
142024-09-03T23:56:55.968ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152024-09-03T23:56:55.981ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_96307640-3db4-4a6e-b03a-71a80f6a607d/crucible/regions/001e5a02-2834-48f1-8d7f-21783d57710a
162024-09-03T23:56:55.981ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_96307640-3db4-4a6e-b03a-71a80f6a607d/crucible/regions/001e5a02-2834-48f1-8d7f-21783d57710a
172024-09-03T23:56:56.030ZINFOcrucible-agent (worker): creating region Region { id: RegionId("001e5a02-2834-48f1-8d7f-21783d57710a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/001e5a02-2834-48f1-8d7f-21783d57710a" region = 001e5a02-2834-48f1-8d7f-21783d57710a
182024-09-03T23:56:56.033ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "001e5a02-2834-48f1-8d7f-21783d57710a" "--data" "/data/regions/001e5a02-2834-48f1-8d7f-21783d57710a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 001e5a02-2834-48f1-8d7f-21783d57710a
192024-09-03T23:56:56.323ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:37855 req_id = 58273f09-f6a4-461d-9538-c2127d064b8e response_code = 200 uri = /crucible/0/regions
202024-09-03T23:56:56.334ZINFOcrucible-agent (worker): region files created ok region = 001e5a02-2834-48f1-8d7f-21783d57710a
212024-09-03T23:56:56.334ZINFOcrucible-agent (datafile): region 001e5a02-2834-48f1-8d7f-21783d57710a state: Requested -> Created
222024-09-03T23:56:56.334ZINFOcrucible-agent (worker): applying SMF actions post create...
232024-09-03T23:56:56.335ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-001e5a02-2834-48f1-8d7f-21783d57710a
242024-09-03T23:56:56.339ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-001e5a02-2834-48f1-8d7f-21783d57710a
252024-09-03T23:56:56.358ZINFOcrucible-agent (worker): creating config property group
262024-09-03T23:56:56.361ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-001e5a02-2834-48f1-8d7f-21783d57710a
272024-09-03T23:56:56.372ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/001e5a02-2834-48f1-8d7f-21783d57710a
282024-09-03T23:56:56.375ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-09-03T23:56:56.375ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
302024-09-03T23:56:56.375ZINFOcrucible-agent (worker): commit
312024-09-03T23:56:56.379ZINFOcrucible-agent (worker): ok!
322024-09-03T23:56:56.385ZINFOcrucible-agent (worker): SMF ok!
332024-09-03T23:56:56.780ZINFOcrucible-agent (dropshot): request completed latency_us = 64 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:37855 req_id = a064f131-727b-461c-9864-69e3c2839dc2 response_code = 200 uri = /crucible/0/regions
342024-09-03T23:57:01.984ZINFOcrucible-agent (datafile): region 4fce7256-f17f-4135-a64a-368d23d02930 state: Requested
352024-09-03T23:57:01.984ZINFOcrucible-agent (dropshot): request completed latency_us = 235 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:37855 req_id = d2dd5690-574b-427d-a056-06ca1c24d756 response_code = 200 uri = /crucible/0/regions
362024-09-03T23:57:01.984ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
372024-09-03T23:57:01.994ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_96307640-3db4-4a6e-b03a-71a80f6a607d/crucible/regions/4fce7256-f17f-4135-a64a-368d23d02930
382024-09-03T23:57:01.994ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_96307640-3db4-4a6e-b03a-71a80f6a607d/crucible/regions/4fce7256-f17f-4135-a64a-368d23d02930
392024-09-03T23:57:02.041ZINFOcrucible-agent (worker): creating region Region { id: RegionId("4fce7256-f17f-4135-a64a-368d23d02930"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/4fce7256-f17f-4135-a64a-368d23d02930" region = 4fce7256-f17f-4135-a64a-368d23d02930
402024-09-03T23:57:02.043ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "4fce7256-f17f-4135-a64a-368d23d02930" "--data" "/data/regions/4fce7256-f17f-4135-a64a-368d23d02930" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 4fce7256-f17f-4135-a64a-368d23d02930
412024-09-03T23:57:02.116ZINFOcrucible-agent (worker): region files created ok region = 4fce7256-f17f-4135-a64a-368d23d02930
422024-09-03T23:57:02.116ZINFOcrucible-agent (datafile): region 4fce7256-f17f-4135-a64a-368d23d02930 state: Requested -> Created
432024-09-03T23:57:02.116ZINFOcrucible-agent (worker): applying SMF actions post create...
442024-09-03T23:57:02.117ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/001e5a02-2834-48f1-8d7f-21783d57710a
452024-09-03T23:57:02.117ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
462024-09-03T23:57:02.117ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
472024-09-03T23:57:02.117ZINFOcrucible-agent (worker): reconfig required
482024-09-03T23:57:02.117ZINFOcrucible-agent (worker): using existing config property group
492024-09-03T23:57:02.117ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-001e5a02-2834-48f1-8d7f-21783d57710a
502024-09-03T23:57:02.117ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/001e5a02-2834-48f1-8d7f-21783d57710a
512024-09-03T23:57:02.120ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:37855 req_id = a39b5cff-fb5d-4a03-beb3-67c31692a621 response_code = 200 uri = /crucible/0/regions
522024-09-03T23:57:02.120ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
532024-09-03T23:57:02.120ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
542024-09-03T23:57:02.120ZINFOcrucible-agent (worker): commit
552024-09-03T23:57:02.125ZINFOcrucible-agent (worker): ok!
562024-09-03T23:57:02.125ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-4fce7256-f17f-4135-a64a-368d23d02930
572024-09-03T23:57:02.128ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-4fce7256-f17f-4135-a64a-368d23d02930
582024-09-03T23:57:02.136ZINFOcrucible-agent (worker): creating config property group
592024-09-03T23:57:02.140ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4fce7256-f17f-4135-a64a-368d23d02930
602024-09-03T23:57:02.140ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4fce7256-f17f-4135-a64a-368d23d02930
612024-09-03T23:57:02.140ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
622024-09-03T23:57:02.140ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
632024-09-03T23:57:02.140ZINFOcrucible-agent (worker): commit
642024-09-03T23:57:02.143ZINFOcrucible-agent (worker): ok!
652024-09-03T23:57:02.147ZINFOcrucible-agent (worker): SMF ok!
662024-09-03T23:57:27.244ZINFOcrucible-agent (dropshot): request completed latency_us = 588 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:37855 req_id = b0d69a08-2a61-41aa-acee-13b5478b52ad response_code = 200 uri = /crucible/0/regions/4fce7256-f17f-4135-a64a-368d23d02930
672024-09-03T23:57:27.248ZINFOcrucible-agent (datafile): path is "/data/regions/4fce7256-f17f-4135-a64a-368d23d02930"
682024-09-03T23:57:27.259ZINFOcrucible-agent (datafile): dataset is oxp_96307640-3db4-4a6e-b03a-71a80f6a607d/crucible/regions/4fce7256-f17f-4135-a64a-368d23d02930
692024-09-03T23:57:27.267ZINFOcrucible-agent (datafile): region 4fce7256-f17f-4135-a64a-368d23d02930 state: Created -> Tombstoned
702024-09-03T23:57:27.267ZINFOcrucible-agent (worker): applying SMF actions before removal...
712024-09-03T23:57:27.267ZINFOcrucible-agent (dropshot): request completed latency_us = 18782 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:37855 req_id = 2d7ec8a0-aed5-45b5-8141-1e5ed608781e response_code = 204 uri = /crucible/0/regions/4fce7256-f17f-4135-a64a-368d23d02930
722024-09-03T23:57:27.268ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-4fce7256-f17f-4135-a64a-368d23d02930 (instance states: (Some(Online), None))
732024-09-03T23:57:27.272ZINFOcrucible-agent (dropshot): request completed latency_us = 605 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:37855 req_id = 5e3cea7f-4d60-42d4-9867-c9421564727c response_code = 200 uri = /crucible/0/regions/4fce7256-f17f-4135-a64a-368d23d02930
742024-09-03T23:57:27.273ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/001e5a02-2834-48f1-8d7f-21783d57710a
752024-09-03T23:57:27.273ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
762024-09-03T23:57:27.273ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
772024-09-03T23:57:27.273ZINFOcrucible-agent (worker): reconfig required
782024-09-03T23:57:27.273ZINFOcrucible-agent (worker): using existing config property group
792024-09-03T23:57:27.273ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-001e5a02-2834-48f1-8d7f-21783d57710a
802024-09-03T23:57:27.273ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/001e5a02-2834-48f1-8d7f-21783d57710a
812024-09-03T23:57:27.273ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
822024-09-03T23:57:27.273ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
832024-09-03T23:57:27.274ZINFOcrucible-agent (worker): commit
842024-09-03T23:57:27.278ZINFOcrucible-agent (worker): ok!
852024-09-03T23:57:27.278ZINFOcrucible-agent (worker): SMF ok!
862024-09-03T23:57:27.288ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_96307640-3db4-4a6e-b03a-71a80f6a607d/crucible/regions/4fce7256-f17f-4135-a64a-368d23d02930" region = 4fce7256-f17f-4135-a64a-368d23d02930
872024-09-03T23:57:27.331ZINFOcrucible-agent (datafile): region 4fce7256-f17f-4135-a64a-368d23d02930 state: Tombstoned -> Destroyed
882024-09-03T23:57:27.384ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:37855 req_id = 5b09f548-3013-485a-8996-d6f585d5d30c response_code = 200 uri = /crucible/0/regions/4fce7256-f17f-4135-a64a-368d23d02930