Max level shown:
1 [ Jul 10 23:01:41 Enabled. ]
2 [ Jul 10 23:01:41 Rereading configuration. ]
3 [ Jul 10 23:01:42 Rereading configuration. ]
4 [ Jul 10 23:01:44 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible -l [fd00:1122:3344:101::14]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-07-10T23:01:45.047ZINFOcrucible-agent: dataset: "oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible"
72024-07-10T23:01:45.052ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::14]:32345
82024-07-10T23:01:45.052ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-07-10T23:01:45.074ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-07-10T23:01:45.203ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::14]:32345
112024-07-10T23:02:03.642ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:43807
122024-07-10T23:02:03.647ZINFOcrucible-agent (datafile): region be092782-558f-47b3-b90b-d3d52191ef49 state: Requested
132024-07-10T23:02:03.653ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
142024-07-10T23:02:03.653ZINFOcrucible-agent (dropshot): request completed latency_us = 5336 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43807 req_id = b6619de0-4a66-4b3f-9381-11ce81911a54 response_code = 200 uri = /crucible/0/regions
152024-07-10T23:02:03.665ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible/regions/be092782-558f-47b3-b90b-d3d52191ef49
162024-07-10T23:02:03.665ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible/regions/be092782-558f-47b3-b90b-d3d52191ef49
172024-07-10T23:02:03.729ZINFOcrucible-agent (worker): creating region Region { id: RegionId("be092782-558f-47b3-b90b-d3d52191ef49"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/be092782-558f-47b3-b90b-d3d52191ef49" region = be092782-558f-47b3-b90b-d3d52191ef49
182024-07-10T23:02:03.730ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "be092782-558f-47b3-b90b-d3d52191ef49" "--data" "/data/regions/be092782-558f-47b3-b90b-d3d52191ef49" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = be092782-558f-47b3-b90b-d3d52191ef49
192024-07-10T23:02:03.838ZINFOcrucible-agent (worker): region files created ok region = be092782-558f-47b3-b90b-d3d52191ef49
202024-07-10T23:02:03.838ZINFOcrucible-agent (datafile): region be092782-558f-47b3-b90b-d3d52191ef49 state: Requested -> Created
212024-07-10T23:02:03.838ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-07-10T23:02:03.838ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-be092782-558f-47b3-b90b-d3d52191ef49
232024-07-10T23:02:03.843ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-be092782-558f-47b3-b90b-d3d52191ef49
242024-07-10T23:02:03.855ZINFOcrucible-agent (worker): creating config property group
252024-07-10T23:02:03.858ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-be092782-558f-47b3-b90b-d3d52191ef49
262024-07-10T23:02:03.858ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/be092782-558f-47b3-b90b-d3d52191ef49
272024-07-10T23:02:03.860ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-07-10T23:02:03.860ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
292024-07-10T23:02:03.860ZINFOcrucible-agent (worker): commit
302024-07-10T23:02:03.867ZINFOcrucible-agent (worker): ok!
312024-07-10T23:02:03.874ZINFOcrucible-agent (worker): SMF ok!
322024-07-10T23:02:03.896ZINFOcrucible-agent (dropshot): request completed latency_us = 79 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43807 req_id = 380995a6-2a3d-4653-8f95-77dc92566de8 response_code = 200 uri = /crucible/0/regions
332024-07-10T23:02:04.869ZINFOcrucible-agent (dropshot): request completed latency_us = 91 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:43807 req_id = 965d36bb-3c29-4de0-82ff-ff3b15e9d5bc response_code = 200 uri = /crucible/0/regions/be092782-558f-47b3-b90b-d3d52191ef49
342024-07-10T23:02:04.873ZINFOcrucible-agent (datafile): path is "/data/regions/be092782-558f-47b3-b90b-d3d52191ef49"
352024-07-10T23:02:04.882ZINFOcrucible-agent (datafile): dataset is oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible/regions/be092782-558f-47b3-b90b-d3d52191ef49
362024-07-10T23:02:04.891ZINFOcrucible-agent (datafile): region be092782-558f-47b3-b90b-d3d52191ef49 state: Created -> Tombstoned
372024-07-10T23:02:04.891ZINFOcrucible-agent (worker): applying SMF actions before removal...
382024-07-10T23:02:04.891ZINFOcrucible-agent (dropshot): request completed latency_us = 18275 local_addr = [fd00:1122:3344:101::14]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:43807 req_id = 54b575d3-9fe0-463c-89c9-9a291173a778 response_code = 204 uri = /crucible/0/regions/be092782-558f-47b3-b90b-d3d52191ef49
392024-07-10T23:02:04.892ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-be092782-558f-47b3-b90b-d3d52191ef49 (instance states: (Some(Online), None))
402024-07-10T23:02:04.894ZINFOcrucible-agent (dropshot): request completed latency_us = 22 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:43807 req_id = 51239b8a-ce18-4ad9-8357-7003d1b3324e response_code = 200 uri = /crucible/0/regions/be092782-558f-47b3-b90b-d3d52191ef49
412024-07-10T23:02:04.897ZINFOcrucible-agent (worker): SMF ok!
422024-07-10T23:02:04.906ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible/regions/be092782-558f-47b3-b90b-d3d52191ef49" region = be092782-558f-47b3-b90b-d3d52191ef49
432024-07-10T23:02:04.972ZINFOcrucible-agent (datafile): region be092782-558f-47b3-b90b-d3d52191ef49 state: Tombstoned -> Destroyed
442024-07-10T23:02:05.036ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:43807 req_id = 800a3ce1-5438-4982-aca8-5c104914aa4d response_code = 200 uri = /crucible/0/regions/be092782-558f-47b3-b90b-d3d52191ef49
452024-07-10T23:02:06.692ZINFOcrucible-agent (datafile): region a8a25c6f-c21a-469d-91b6-bf6fdbca5c33 state: Requested
462024-07-10T23:02:06.694ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
472024-07-10T23:02:06.694ZINFOcrucible-agent (dropshot): request completed latency_us = 2993 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43807 req_id = 449c7184-72f8-4cb3-bc25-aad8106d2fb4 response_code = 200 uri = /crucible/0/regions
482024-07-10T23:02:06.704ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33
492024-07-10T23:02:06.704ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33
502024-07-10T23:02:06.780ZINFOcrucible-agent (worker): creating region Region { id: RegionId("a8a25c6f-c21a-469d-91b6-bf6fdbca5c33"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33" region = a8a25c6f-c21a-469d-91b6-bf6fdbca5c33
512024-07-10T23:02:06.783ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "a8a25c6f-c21a-469d-91b6-bf6fdbca5c33" "--data" "/data/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = a8a25c6f-c21a-469d-91b6-bf6fdbca5c33
522024-07-10T23:02:06.839ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43807 req_id = 93de4e60-7cce-44d2-b4c2-d4a7d4ba91d1 response_code = 200 uri = /crucible/0/regions
532024-07-10T23:02:06.848ZINFOcrucible-agent (worker): region files created ok region = a8a25c6f-c21a-469d-91b6-bf6fdbca5c33
542024-07-10T23:02:06.848ZINFOcrucible-agent (datafile): region a8a25c6f-c21a-469d-91b6-bf6fdbca5c33 state: Requested -> Created
552024-07-10T23:02:06.848ZINFOcrucible-agent (worker): applying SMF actions post create...
562024-07-10T23:02:06.849ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-a8a25c6f-c21a-469d-91b6-bf6fdbca5c33
572024-07-10T23:02:06.852ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-a8a25c6f-c21a-469d-91b6-bf6fdbca5c33
582024-07-10T23:02:06.860ZINFOcrucible-agent (worker): creating config property group
592024-07-10T23:02:06.864ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-a8a25c6f-c21a-469d-91b6-bf6fdbca5c33
602024-07-10T23:02:06.864ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33
612024-07-10T23:02:06.864ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
622024-07-10T23:02:06.864ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
632024-07-10T23:02:06.864ZINFOcrucible-agent (worker): commit
642024-07-10T23:02:06.868ZINFOcrucible-agent (worker): ok!
652024-07-10T23:02:06.873ZINFOcrucible-agent (worker): SMF ok!
662024-07-10T23:02:07.375ZINFOcrucible-agent (dropshot): request completed latency_us = 49 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43807 req_id = f23c563a-9732-43c4-9f36-31d55e094fe0 response_code = 200 uri = /crucible/0/regions
672024-07-10T23:07:25.068ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:61224
682024-07-10T23:07:25.068ZINFOcrucible-agent (datafile): region 07603ca6-66fd-4954-82a9-a4fb3b6298eb state: Requested
692024-07-10T23:07:25.069ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
702024-07-10T23:07:25.069ZINFOcrucible-agent (dropshot): request completed latency_us = 789 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61224 req_id = e737ca91-ef76-4e4b-b7b0-5cfb264999e0 response_code = 200 uri = /crucible/0/regions
712024-07-10T23:07:25.079ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible/regions/07603ca6-66fd-4954-82a9-a4fb3b6298eb
722024-07-10T23:07:25.079ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible/regions/07603ca6-66fd-4954-82a9-a4fb3b6298eb
732024-07-10T23:07:25.164ZINFOcrucible-agent (worker): creating region Region { id: RegionId("07603ca6-66fd-4954-82a9-a4fb3b6298eb"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/07603ca6-66fd-4954-82a9-a4fb3b6298eb" region = 07603ca6-66fd-4954-82a9-a4fb3b6298eb
742024-07-10T23:07:25.164ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "07603ca6-66fd-4954-82a9-a4fb3b6298eb" "--data" "/data/regions/07603ca6-66fd-4954-82a9-a4fb3b6298eb" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 07603ca6-66fd-4954-82a9-a4fb3b6298eb
752024-07-10T23:07:25.249ZINFOcrucible-agent (worker): region files created ok region = 07603ca6-66fd-4954-82a9-a4fb3b6298eb
762024-07-10T23:07:25.249ZINFOcrucible-agent (datafile): region 07603ca6-66fd-4954-82a9-a4fb3b6298eb state: Requested -> Created
772024-07-10T23:07:25.249ZINFOcrucible-agent (worker): applying SMF actions post create...
782024-07-10T23:07:25.250ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-07603ca6-66fd-4954-82a9-a4fb3b6298eb
792024-07-10T23:07:25.259ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-07603ca6-66fd-4954-82a9-a4fb3b6298eb
802024-07-10T23:07:25.273ZINFOcrucible-agent (worker): creating config property group
812024-07-10T23:07:25.277ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-07603ca6-66fd-4954-82a9-a4fb3b6298eb
822024-07-10T23:07:25.277ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/07603ca6-66fd-4954-82a9-a4fb3b6298eb
832024-07-10T23:07:25.277ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
842024-07-10T23:07:25.277ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
852024-07-10T23:07:25.277ZINFOcrucible-agent (worker): commit
862024-07-10T23:07:25.282ZINFOcrucible-agent (worker): ok!
872024-07-10T23:07:25.289ZINFOcrucible-agent (worker): SMF ok!
882024-07-10T23:07:25.349ZINFOcrucible-agent (dropshot): request completed latency_us = 77 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61224 req_id = 5fbc9b8e-81d6-4974-bc13-0287335d3314 response_code = 200 uri = /crucible/0/regions
892024-07-10T23:07:26.830ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:36518
902024-07-10T23:07:26.831ZINFOcrucible-agent (dropshot): request completed latency_us = 707 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36518 req_id = 143ba143-ec6e-41dc-9094-4fe472ae3535 response_code = 200 uri = /crucible/0/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33
912024-07-10T23:07:26.831ZINFOcrucible-agent (datafile): path is "/data/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33"
922024-07-10T23:07:26.841ZINFOcrucible-agent (datafile): dataset is oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33
932024-07-10T23:07:26.850ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33@9afadc7e-5ad8-44de-833e-b82e207e996e
942024-07-10T23:07:26.850ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33", "9afadc7e-5ad8-44de-833e-b82e207e996e"]
952024-07-10T23:07:26.862ZINFOcrucible-agent (dropshot): request completed latency_us = 30831 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36518 req_id = 8d7807a2-485c-4469-8035-f1e4be045827 response_code = 200 uri = /crucible/0/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33/snapshots/9afadc7e-5ad8-44de-833e-b82e207e996e
962024-07-10T23:07:26.862ZINFOcrucible-agent (datafile): path is "/data/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33"
972024-07-10T23:07:26.871ZINFOcrucible-agent (datafile): dataset is oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33
982024-07-10T23:07:26.879ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33@9afadc7e-5ad8-44de-833e-b82e207e996e
992024-07-10T23:07:26.879ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_68327ee5-4aec-4399-b8bb-563ba202e23d/crucible/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33", "9afadc7e-5ad8-44de-833e-b82e207e996e"]
1002024-07-10T23:07:26.891ZINFOcrucible-agent (datafile): requesting running snapshot a8a25c6f-c21a-469d-91b6-bf6fdbca5c33-9afadc7e-5ad8-44de-833e-b82e207e996e state: Requested
1012024-07-10T23:07:26.891ZINFOcrucible-agent (worker): applying SMF actions for region a8a25c6f-c21a-469d-91b6-bf6fdbca5c33 running snapshot 9afadc7e-5ad8-44de-833e-b82e207e996e (state Requested)...
1022024-07-10T23:07:26.891ZINFOcrucible-agent (dropshot): request completed latency_us = 29311 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36518 req_id = ce2e1cc4-4458-4adc-b811-5407ab2bc50f response_code = 200 uri = /crucible/0/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33/snapshots/9afadc7e-5ad8-44de-833e-b82e207e996e/run
1032024-07-10T23:07:26.894ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-a8a25c6f-c21a-469d-91b6-bf6fdbca5c33-9afadc7e-5ad8-44de-833e-b82e207e996e
1042024-07-10T23:07:26.897ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-a8a25c6f-c21a-469d-91b6-bf6fdbca5c33-9afadc7e-5ad8-44de-833e-b82e207e996e
1052024-07-10T23:07:26.907ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-a8a25c6f-c21a-469d-91b6-bf6fdbca5c33-9afadc7e-5ad8-44de-833e-b82e207e996e
1062024-07-10T23:07:26.907ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a8a25c6f-c21a-469d-91b6-bf6fdbca5c33/.zfs/snapshot/9afadc7e-5ad8-44de-833e-b82e207e996e
1072024-07-10T23:07:26.907ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1082024-07-10T23:07:26.907ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
1092024-07-10T23:07:26.907ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
1102024-07-10T23:07:26.907ZINFOcrucible-agent (worker): commit
1112024-07-10T23:07:26.911ZINFOcrucible-agent (worker): ok!
1122024-07-10T23:07:26.915ZINFOcrucible-agent (worker): SMF ok!
1132024-07-10T23:07:26.915ZINFOcrucible-agent (datafile): region a8a25c6f-c21a-469d-91b6-bf6fdbca5c33 running snapshot 9afadc7e-5ad8-44de-833e-b82e207e996e state: Requested -> Created