Max level shown:
1 [ Aug 3 04:36:44 Enabled. ]
2 [ Aug 3 04:36:44 Rereading configuration. ]
3 [ Aug 3 04:36:50 Rereading configuration. ]
4 [ Aug 3 04:36:54 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible -l [fd00:1122:3344:101::17]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-08-03T04:36:54.507ZINFOcrucible-agent: dataset: "oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible"
72024-08-03T04:36:54.508ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::17]:32345
82024-08-03T04:36:54.508ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-08-03T04:36:54.526ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-08-03T04:36:54.638ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::17]:32345
112024-08-03T04:37:49.175ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::c]:42829
122024-08-03T04:37:49.195ZINFOcrucible-agent (datafile): region 6e25da06-99c3-4fcc-8b45-8118f931c9ce state: Requested
132024-08-03T04:37:49.201ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
142024-08-03T04:37:49.205ZINFOcrucible-agent (dropshot): request completed latency_us = 16395 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:42829 req_id = c75e5cbf-3ee9-492d-a3f1-c656e48bfdf5 response_code = 200 uri = /crucible/0/regions
152024-08-03T04:37:49.212ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible/regions/6e25da06-99c3-4fcc-8b45-8118f931c9ce
162024-08-03T04:37:49.212ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible/regions/6e25da06-99c3-4fcc-8b45-8118f931c9ce
172024-08-03T04:37:49.272ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6e25da06-99c3-4fcc-8b45-8118f931c9ce"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/6e25da06-99c3-4fcc-8b45-8118f931c9ce" region = 6e25da06-99c3-4fcc-8b45-8118f931c9ce
182024-08-03T04:37:49.274ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "6e25da06-99c3-4fcc-8b45-8118f931c9ce" "--data" "/data/regions/6e25da06-99c3-4fcc-8b45-8118f931c9ce" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 6e25da06-99c3-4fcc-8b45-8118f931c9ce
192024-08-03T04:37:49.393ZINFOcrucible-agent (worker): region files created ok region = 6e25da06-99c3-4fcc-8b45-8118f931c9ce
202024-08-03T04:37:49.393ZINFOcrucible-agent (datafile): region 6e25da06-99c3-4fcc-8b45-8118f931c9ce state: Requested -> Created
212024-08-03T04:37:49.393ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-08-03T04:37:49.394ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6e25da06-99c3-4fcc-8b45-8118f931c9ce
232024-08-03T04:37:49.400ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6e25da06-99c3-4fcc-8b45-8118f931c9ce
242024-08-03T04:37:49.417ZINFOcrucible-agent (worker): creating config property group
252024-08-03T04:37:49.421ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6e25da06-99c3-4fcc-8b45-8118f931c9ce
262024-08-03T04:37:49.430ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6e25da06-99c3-4fcc-8b45-8118f931c9ce
272024-08-03T04:37:49.432ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-08-03T04:37:49.432ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
292024-08-03T04:37:49.432ZINFOcrucible-agent (worker): commit
302024-08-03T04:37:49.437ZINFOcrucible-agent (worker): ok!
312024-08-03T04:37:49.441ZINFOcrucible-agent (worker): SMF ok!
322024-08-03T04:37:49.556ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:42829 req_id = f85e3e63-9d84-4a2b-80b9-4903d99af633 response_code = 200 uri = /crucible/0/regions
332024-08-03T04:37:50.424ZINFOcrucible-agent (dropshot): request completed latency_us = 596 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:42829 req_id = c634e497-3b71-4351-9244-a80dcb927d5b response_code = 200 uri = /crucible/0/regions/6e25da06-99c3-4fcc-8b45-8118f931c9ce
342024-08-03T04:37:50.426ZINFOcrucible-agent (datafile): path is "/data/regions/6e25da06-99c3-4fcc-8b45-8118f931c9ce"
352024-08-03T04:37:50.436ZINFOcrucible-agent (datafile): dataset is oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible/regions/6e25da06-99c3-4fcc-8b45-8118f931c9ce
362024-08-03T04:37:50.445ZINFOcrucible-agent (datafile): region 6e25da06-99c3-4fcc-8b45-8118f931c9ce state: Created -> Tombstoned
372024-08-03T04:37:50.445ZINFOcrucible-agent (worker): applying SMF actions before removal...
382024-08-03T04:37:50.445ZINFOcrucible-agent (dropshot): request completed latency_us = 19074 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:42829 req_id = 7b63d09a-fd51-4504-8304-dd7af93e507a response_code = 204 uri = /crucible/0/regions/6e25da06-99c3-4fcc-8b45-8118f931c9ce
392024-08-03T04:37:50.446ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-6e25da06-99c3-4fcc-8b45-8118f931c9ce (instance states: (Some(Online), None))
402024-08-03T04:37:50.449ZINFOcrucible-agent (dropshot): request completed latency_us = 24 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:42829 req_id = 457c73ed-7e1f-4d1f-8ad0-d4d270816596 response_code = 200 uri = /crucible/0/regions/6e25da06-99c3-4fcc-8b45-8118f931c9ce
412024-08-03T04:37:50.451ZINFOcrucible-agent (worker): SMF ok!
422024-08-03T04:37:50.461ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible/regions/6e25da06-99c3-4fcc-8b45-8118f931c9ce" region = 6e25da06-99c3-4fcc-8b45-8118f931c9ce
432024-08-03T04:37:50.515ZINFOcrucible-agent (datafile): region 6e25da06-99c3-4fcc-8b45-8118f931c9ce state: Tombstoned -> Destroyed
442024-08-03T04:37:50.554ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:42829 req_id = 470fea74-7d94-45e0-83d6-b71f2eece930 response_code = 200 uri = /crucible/0/regions/6e25da06-99c3-4fcc-8b45-8118f931c9ce
452024-08-03T04:38:09.408ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:41527
462024-08-03T04:38:09.408ZINFOcrucible-agent (datafile): region 0486e619-bc51-40b8-af53-a76431b1b748 state: Requested
472024-08-03T04:38:09.409ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
482024-08-03T04:38:09.409ZINFOcrucible-agent (dropshot): request completed latency_us = 209 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:41527 req_id = f01c082e-bb4f-4173-9d5a-5d1ab52568a4 response_code = 200 uri = /crucible/0/regions
492024-08-03T04:38:09.418ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible/regions/0486e619-bc51-40b8-af53-a76431b1b748
502024-08-03T04:38:09.418ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible/regions/0486e619-bc51-40b8-af53-a76431b1b748
512024-08-03T04:38:09.471ZINFOcrucible-agent (worker): creating region Region { id: RegionId("0486e619-bc51-40b8-af53-a76431b1b748"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/0486e619-bc51-40b8-af53-a76431b1b748" region = 0486e619-bc51-40b8-af53-a76431b1b748
522024-08-03T04:38:09.471ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "0486e619-bc51-40b8-af53-a76431b1b748" "--data" "/data/regions/0486e619-bc51-40b8-af53-a76431b1b748" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 0486e619-bc51-40b8-af53-a76431b1b748
532024-08-03T04:38:09.551ZINFOcrucible-agent (worker): region files created ok region = 0486e619-bc51-40b8-af53-a76431b1b748
542024-08-03T04:38:09.551ZINFOcrucible-agent (datafile): region 0486e619-bc51-40b8-af53-a76431b1b748 state: Requested -> Created
552024-08-03T04:38:09.551ZINFOcrucible-agent (worker): applying SMF actions post create...
562024-08-03T04:38:09.552ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-0486e619-bc51-40b8-af53-a76431b1b748
572024-08-03T04:38:09.556ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-0486e619-bc51-40b8-af53-a76431b1b748
582024-08-03T04:38:09.564ZINFOcrucible-agent (worker): creating config property group
592024-08-03T04:38:09.568ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0486e619-bc51-40b8-af53-a76431b1b748
602024-08-03T04:38:09.568ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0486e619-bc51-40b8-af53-a76431b1b748
612024-08-03T04:38:09.569ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
622024-08-03T04:38:09.569ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
632024-08-03T04:38:09.569ZINFOcrucible-agent (worker): commit
642024-08-03T04:38:09.573ZINFOcrucible-agent (worker): ok!
652024-08-03T04:38:09.578ZINFOcrucible-agent (worker): SMF ok!
662024-08-03T04:38:09.676ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:41527 req_id = 6510ff56-9bfd-44e3-83a2-893ce5e30217 response_code = 200 uri = /crucible/0/regions
672024-08-03T04:43:37.035ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:58547
682024-08-03T04:43:37.035ZINFOcrucible-agent (datafile): region 1aad207d-42bb-48bc-982d-e41b634c8f3f state: Requested
692024-08-03T04:43:37.035ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
702024-08-03T04:43:37.035ZINFOcrucible-agent (dropshot): request completed latency_us = 657 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:58547 req_id = 9f768660-472f-4cee-a449-53ff5d8b2444 response_code = 200 uri = /crucible/0/regions
712024-08-03T04:43:37.046ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible/regions/1aad207d-42bb-48bc-982d-e41b634c8f3f
722024-08-03T04:43:37.046ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible/regions/1aad207d-42bb-48bc-982d-e41b634c8f3f
732024-08-03T04:43:37.103ZINFOcrucible-agent (worker): creating region Region { id: RegionId("1aad207d-42bb-48bc-982d-e41b634c8f3f"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/1aad207d-42bb-48bc-982d-e41b634c8f3f" region = 1aad207d-42bb-48bc-982d-e41b634c8f3f
742024-08-03T04:43:37.103ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "1aad207d-42bb-48bc-982d-e41b634c8f3f" "--data" "/data/regions/1aad207d-42bb-48bc-982d-e41b634c8f3f" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 1aad207d-42bb-48bc-982d-e41b634c8f3f
752024-08-03T04:43:37.182ZINFOcrucible-agent (worker): region files created ok region = 1aad207d-42bb-48bc-982d-e41b634c8f3f
762024-08-03T04:43:37.182ZINFOcrucible-agent (datafile): region 1aad207d-42bb-48bc-982d-e41b634c8f3f state: Requested -> Created
772024-08-03T04:43:37.182ZINFOcrucible-agent (worker): applying SMF actions post create...
782024-08-03T04:43:37.189ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-1aad207d-42bb-48bc-982d-e41b634c8f3f
792024-08-03T04:43:37.196ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-1aad207d-42bb-48bc-982d-e41b634c8f3f
802024-08-03T04:43:37.210ZINFOcrucible-agent (worker): creating config property group
812024-08-03T04:43:37.214ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-1aad207d-42bb-48bc-982d-e41b634c8f3f
822024-08-03T04:43:37.214ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1aad207d-42bb-48bc-982d-e41b634c8f3f
832024-08-03T04:43:37.214ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
842024-08-03T04:43:37.214ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
852024-08-03T04:43:37.214ZINFOcrucible-agent (worker): commit
862024-08-03T04:43:37.221ZINFOcrucible-agent (worker): ok!
872024-08-03T04:43:37.227ZINFOcrucible-agent (worker): SMF ok!
882024-08-03T04:43:37.319ZINFOcrucible-agent (dropshot): request completed latency_us = 115 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:58547 req_id = 51c3b6a4-1a50-4db7-86db-b37cb0ccad0e response_code = 200 uri = /crucible/0/regions
892024-08-03T04:43:38.066ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:53317
902024-08-03T04:43:38.067ZINFOcrucible-agent (dropshot): request completed latency_us = 1000 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:53317 req_id = f455a853-af7f-4475-88e2-758f9905d1ed response_code = 200 uri = /crucible/0/regions/0486e619-bc51-40b8-af53-a76431b1b748
912024-08-03T04:43:38.070ZINFOcrucible-agent (datafile): path is "/data/regions/0486e619-bc51-40b8-af53-a76431b1b748"
922024-08-03T04:43:38.080ZINFOcrucible-agent (datafile): dataset is oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible/regions/0486e619-bc51-40b8-af53-a76431b1b748
932024-08-03T04:43:38.092ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible/regions/0486e619-bc51-40b8-af53-a76431b1b748@b65cb9c8-ff1b-4ef9-a79a-705765fb831b
942024-08-03T04:43:38.092ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible/regions/0486e619-bc51-40b8-af53-a76431b1b748", "b65cb9c8-ff1b-4ef9-a79a-705765fb831b"]
952024-08-03T04:43:38.103ZINFOcrucible-agent (dropshot): request completed latency_us = 36033 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:53317 req_id = 5203bd7b-a62d-40a0-94f6-d0b8ad6bf7c1 response_code = 200 uri = /crucible/0/regions/0486e619-bc51-40b8-af53-a76431b1b748/snapshots/b65cb9c8-ff1b-4ef9-a79a-705765fb831b
962024-08-03T04:43:38.105ZINFOcrucible-agent (datafile): path is "/data/regions/0486e619-bc51-40b8-af53-a76431b1b748"
972024-08-03T04:43:38.113ZINFOcrucible-agent (datafile): dataset is oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible/regions/0486e619-bc51-40b8-af53-a76431b1b748
982024-08-03T04:43:38.122ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible/regions/0486e619-bc51-40b8-af53-a76431b1b748@b65cb9c8-ff1b-4ef9-a79a-705765fb831b
992024-08-03T04:43:38.122ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_ec22abdc-76d0-4dce-9241-6d7acac56c81/crucible/regions/0486e619-bc51-40b8-af53-a76431b1b748", "b65cb9c8-ff1b-4ef9-a79a-705765fb831b"]
1002024-08-03T04:43:38.132ZINFOcrucible-agent (datafile): requesting running snapshot 0486e619-bc51-40b8-af53-a76431b1b748-b65cb9c8-ff1b-4ef9-a79a-705765fb831b state: Requested
1012024-08-03T04:43:38.132ZINFOcrucible-agent (worker): applying SMF actions for region 0486e619-bc51-40b8-af53-a76431b1b748 running snapshot b65cb9c8-ff1b-4ef9-a79a-705765fb831b (state Requested)...
1022024-08-03T04:43:38.132ZINFOcrucible-agent (dropshot): request completed latency_us = 27440 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53317 req_id = 732cd848-48f8-4db5-ad00-33d0f5ef56f7 response_code = 200 uri = /crucible/0/regions/0486e619-bc51-40b8-af53-a76431b1b748/snapshots/b65cb9c8-ff1b-4ef9-a79a-705765fb831b/run
1032024-08-03T04:43:38.135ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-0486e619-bc51-40b8-af53-a76431b1b748-b65cb9c8-ff1b-4ef9-a79a-705765fb831b
1042024-08-03T04:43:38.141ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-0486e619-bc51-40b8-af53-a76431b1b748-b65cb9c8-ff1b-4ef9-a79a-705765fb831b
1052024-08-03T04:43:38.159ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-0486e619-bc51-40b8-af53-a76431b1b748-b65cb9c8-ff1b-4ef9-a79a-705765fb831b
1062024-08-03T04:43:38.159ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0486e619-bc51-40b8-af53-a76431b1b748/.zfs/snapshot/b65cb9c8-ff1b-4ef9-a79a-705765fb831b
1072024-08-03T04:43:38.159ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1082024-08-03T04:43:38.160ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
1092024-08-03T04:43:38.160ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
1102024-08-03T04:43:38.160ZINFOcrucible-agent (worker): commit
1112024-08-03T04:43:38.164ZINFOcrucible-agent (worker): ok!
1122024-08-03T04:43:38.169ZINFOcrucible-agent (worker): SMF ok!
1132024-08-03T04:43:38.169ZINFOcrucible-agent (datafile): region 0486e619-bc51-40b8-af53-a76431b1b748 running snapshot b65cb9c8-ff1b-4ef9-a79a-705765fb831b state: Requested -> Created