Max level shown:
1 [ Jul 10 20:02:44 Enabled. ]
2 [ Jul 10 20:02:44 Rereading configuration. ]
3 [ Jul 10 20:02:46 Rereading configuration. ]
4 [ Jul 10 20:02:48 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible -l [fd00:1122:3344:101::19]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-07-10T20:02:48.123ZINFOcrucible-agent: dataset: "oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible"
72024-07-10T20:02:48.124ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::19]:32345
82024-07-10T20:02:48.124ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-07-10T20:02:48.165ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-07-10T20:02:48.324ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::19]:32345
112024-07-10T20:03:13.536ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::c]:50205
122024-07-10T20:03:13.555ZINFOcrucible-agent (datafile): region 54882e6d-646e-4597-bc2b-a9a838f0f21c state: Requested
132024-07-10T20:03:13.561ZINFOcrucible-agent (dropshot): request completed latency_us = 14240 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:50205 req_id = de32ae94-bd64-462b-80bf-e54fd5547453 response_code = 200 uri = /crucible/0/regions
142024-07-10T20:03:13.562ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152024-07-10T20:03:13.575ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible/regions/54882e6d-646e-4597-bc2b-a9a838f0f21c
162024-07-10T20:03:13.575ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible/regions/54882e6d-646e-4597-bc2b-a9a838f0f21c
172024-07-10T20:03:13.668ZINFOcrucible-agent (worker): creating region Region { id: RegionId("54882e6d-646e-4597-bc2b-a9a838f0f21c"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/54882e6d-646e-4597-bc2b-a9a838f0f21c" region = 54882e6d-646e-4597-bc2b-a9a838f0f21c
182024-07-10T20:03:13.670ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "54882e6d-646e-4597-bc2b-a9a838f0f21c" "--data" "/data/regions/54882e6d-646e-4597-bc2b-a9a838f0f21c" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 54882e6d-646e-4597-bc2b-a9a838f0f21c
192024-07-10T20:03:13.875ZINFOcrucible-agent (worker): region files created ok region = 54882e6d-646e-4597-bc2b-a9a838f0f21c
202024-07-10T20:03:13.876ZINFOcrucible-agent (datafile): region 54882e6d-646e-4597-bc2b-a9a838f0f21c state: Requested -> Created
212024-07-10T20:03:13.876ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-07-10T20:03:13.876ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-54882e6d-646e-4597-bc2b-a9a838f0f21c
232024-07-10T20:03:13.896ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-54882e6d-646e-4597-bc2b-a9a838f0f21c
242024-07-10T20:03:13.926ZINFOcrucible-agent (dropshot): request completed latency_us = 93 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:50205 req_id = 24bb26e9-6899-4060-a6ce-9b854ed0e114 response_code = 200 uri = /crucible/0/regions
252024-07-10T20:03:13.941ZINFOcrucible-agent (worker): creating config property group
262024-07-10T20:03:13.958ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-54882e6d-646e-4597-bc2b-a9a838f0f21c
272024-07-10T20:03:13.959ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/54882e6d-646e-4597-bc2b-a9a838f0f21c
282024-07-10T20:03:13.963ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-07-10T20:03:13.963ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
302024-07-10T20:03:13.963ZINFOcrucible-agent (worker): commit
312024-07-10T20:03:13.996ZINFOcrucible-agent (worker): ok!
322024-07-10T20:03:14.018ZINFOcrucible-agent (worker): SMF ok!
332024-07-10T20:03:15.306ZINFOcrucible-agent (dropshot): request completed latency_us = 93 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:50205 req_id = c1f1dede-6ece-469c-8481-75698ca2b3f0 response_code = 200 uri = /crucible/0/regions/54882e6d-646e-4597-bc2b-a9a838f0f21c
342024-07-10T20:03:15.308ZINFOcrucible-agent (datafile): path is "/data/regions/54882e6d-646e-4597-bc2b-a9a838f0f21c"
352024-07-10T20:03:15.317ZINFOcrucible-agent (datafile): dataset is oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible/regions/54882e6d-646e-4597-bc2b-a9a838f0f21c
362024-07-10T20:03:15.327ZINFOcrucible-agent (datafile): region 54882e6d-646e-4597-bc2b-a9a838f0f21c state: Created -> Tombstoned
372024-07-10T20:03:15.327ZINFOcrucible-agent (worker): applying SMF actions before removal...
382024-07-10T20:03:15.327ZINFOcrucible-agent (dropshot): request completed latency_us = 19318 local_addr = [fd00:1122:3344:101::19]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:50205 req_id = 942029e9-853b-44bb-a7ad-5d8ea045315e response_code = 204 uri = /crucible/0/regions/54882e6d-646e-4597-bc2b-a9a838f0f21c
392024-07-10T20:03:15.328ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-54882e6d-646e-4597-bc2b-a9a838f0f21c (instance states: (Some(Online), None))
402024-07-10T20:03:15.329ZINFOcrucible-agent (dropshot): request completed latency_us = 31 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:50205 req_id = eb531e33-bb50-4ccf-8b27-975ae93c5e1c response_code = 200 uri = /crucible/0/regions/54882e6d-646e-4597-bc2b-a9a838f0f21c
412024-07-10T20:03:15.348ZINFOcrucible-agent (worker): SMF ok!
422024-07-10T20:03:15.358ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible/regions/54882e6d-646e-4597-bc2b-a9a838f0f21c" region = 54882e6d-646e-4597-bc2b-a9a838f0f21c
432024-07-10T20:03:15.411ZINFOcrucible-agent (dropshot): request completed latency_us = 83 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:50205 req_id = 46e3f915-a9af-49d9-a3d4-86d021468d43 response_code = 200 uri = /crucible/0/regions/54882e6d-646e-4597-bc2b-a9a838f0f21c
442024-07-10T20:03:15.435ZINFOcrucible-agent (datafile): region 54882e6d-646e-4597-bc2b-a9a838f0f21c state: Tombstoned -> Destroyed
452024-07-10T20:03:15.478ZINFOcrucible-agent (dropshot): request completed latency_us = 31 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:50205 req_id = a3abe02d-5a33-4829-af32-47ab2cd6823e response_code = 200 uri = /crucible/0/regions/54882e6d-646e-4597-bc2b-a9a838f0f21c
462024-07-10T20:04:01.384ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:33501
472024-07-10T20:04:01.384ZINFOcrucible-agent (datafile): region 106fca46-fef1-4b71-bb4b-7b85d6deea47 state: Requested
482024-07-10T20:04:01.387ZINFOcrucible-agent (dropshot): request completed latency_us = 2409 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33501 req_id = dae7fc18-6cb9-429b-aa29-e3af1d8154b0 response_code = 200 uri = /crucible/0/regions
492024-07-10T20:04:01.387ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
502024-07-10T20:04:01.395ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47
512024-07-10T20:04:01.395ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47
522024-07-10T20:04:01.458ZINFOcrucible-agent (worker): creating region Region { id: RegionId("106fca46-fef1-4b71-bb4b-7b85d6deea47"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47" region = 106fca46-fef1-4b71-bb4b-7b85d6deea47
532024-07-10T20:04:01.460ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "106fca46-fef1-4b71-bb4b-7b85d6deea47" "--data" "/data/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 106fca46-fef1-4b71-bb4b-7b85d6deea47
542024-07-10T20:04:01.565ZINFOcrucible-agent (worker): region files created ok region = 106fca46-fef1-4b71-bb4b-7b85d6deea47
552024-07-10T20:04:01.565ZINFOcrucible-agent (datafile): region 106fca46-fef1-4b71-bb4b-7b85d6deea47 state: Requested -> Created
562024-07-10T20:04:01.565ZINFOcrucible-agent (worker): applying SMF actions post create...
572024-07-10T20:04:01.567ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-106fca46-fef1-4b71-bb4b-7b85d6deea47
582024-07-10T20:04:01.573ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-106fca46-fef1-4b71-bb4b-7b85d6deea47
592024-07-10T20:04:01.585ZINFOcrucible-agent (worker): creating config property group
602024-07-10T20:04:01.591ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-106fca46-fef1-4b71-bb4b-7b85d6deea47
612024-07-10T20:04:01.592ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47
622024-07-10T20:04:01.592ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
632024-07-10T20:04:01.592ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
642024-07-10T20:04:01.592ZINFOcrucible-agent (worker): commit
652024-07-10T20:04:01.599ZINFOcrucible-agent (worker): ok!
662024-07-10T20:04:01.604ZINFOcrucible-agent (worker): SMF ok!
672024-07-10T20:04:01.716ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33501 req_id = 74974178-2b87-4900-858d-4ca3e326a149 response_code = 200 uri = /crucible/0/regions
682024-07-10T20:09:21.298ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:39401
692024-07-10T20:09:21.298ZINFOcrucible-agent (datafile): region 43bf9777-3115-43da-80e7-f772951318fd state: Requested
702024-07-10T20:09:21.298ZINFOcrucible-agent (dropshot): request completed latency_us = 813 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39401 req_id = 92bc35ea-7e5d-4440-9d04-2d408c883473 response_code = 200 uri = /crucible/0/regions
712024-07-10T20:09:21.298ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
722024-07-10T20:09:21.308ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible/regions/43bf9777-3115-43da-80e7-f772951318fd
732024-07-10T20:09:21.308ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible/regions/43bf9777-3115-43da-80e7-f772951318fd
742024-07-10T20:09:21.361ZINFOcrucible-agent (worker): creating region Region { id: RegionId("43bf9777-3115-43da-80e7-f772951318fd"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/43bf9777-3115-43da-80e7-f772951318fd" region = 43bf9777-3115-43da-80e7-f772951318fd
752024-07-10T20:09:21.361ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "43bf9777-3115-43da-80e7-f772951318fd" "--data" "/data/regions/43bf9777-3115-43da-80e7-f772951318fd" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 43bf9777-3115-43da-80e7-f772951318fd
762024-07-10T20:09:21.427ZINFOcrucible-agent (worker): region files created ok region = 43bf9777-3115-43da-80e7-f772951318fd
772024-07-10T20:09:21.427ZINFOcrucible-agent (datafile): region 43bf9777-3115-43da-80e7-f772951318fd state: Requested -> Created
782024-07-10T20:09:21.427ZINFOcrucible-agent (worker): applying SMF actions post create...
792024-07-10T20:09:21.434ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-43bf9777-3115-43da-80e7-f772951318fd
802024-07-10T20:09:21.441ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-43bf9777-3115-43da-80e7-f772951318fd
812024-07-10T20:09:21.450ZINFOcrucible-agent (worker): creating config property group
822024-07-10T20:09:21.453ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-43bf9777-3115-43da-80e7-f772951318fd
832024-07-10T20:09:21.453ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/43bf9777-3115-43da-80e7-f772951318fd
842024-07-10T20:09:21.453ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
852024-07-10T20:09:21.453ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
862024-07-10T20:09:21.453ZINFOcrucible-agent (worker): commit
872024-07-10T20:09:21.456ZINFOcrucible-agent (worker): ok!
882024-07-10T20:09:21.460ZINFOcrucible-agent (worker): SMF ok!
892024-07-10T20:09:21.602ZINFOcrucible-agent (dropshot): request completed latency_us = 79 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39401 req_id = 9be30df8-f490-431d-bc4e-04ef039f8b2a response_code = 200 uri = /crucible/0/regions
902024-07-10T20:09:22.482ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:62942
912024-07-10T20:09:22.482ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:62942 req_id = e9b3a38c-8cbd-4cbd-a353-519395d2a645 response_code = 200 uri = /crucible/0/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47
922024-07-10T20:09:22.482ZINFOcrucible-agent (datafile): path is "/data/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47"
932024-07-10T20:09:22.491ZINFOcrucible-agent (datafile): dataset is oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47
942024-07-10T20:09:22.499ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47@9e302fab-288d-4912-8d43-d6946fbfde66
952024-07-10T20:09:22.499ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47", "9e302fab-288d-4912-8d43-d6946fbfde66"]
962024-07-10T20:09:22.510ZINFOcrucible-agent (dropshot): request completed latency_us = 28177 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:62942 req_id = 48eb40b6-b764-4ef9-846d-cd924682d317 response_code = 200 uri = /crucible/0/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47/snapshots/9e302fab-288d-4912-8d43-d6946fbfde66
972024-07-10T20:09:22.510ZINFOcrucible-agent (datafile): path is "/data/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47"
982024-07-10T20:09:22.520ZINFOcrucible-agent (datafile): dataset is oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47
992024-07-10T20:09:22.529ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47@9e302fab-288d-4912-8d43-d6946fbfde66
1002024-07-10T20:09:22.529ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_9c3571c0-266e-418f-b268-1185bbcb3a3c/crucible/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47", "9e302fab-288d-4912-8d43-d6946fbfde66"]
1012024-07-10T20:09:22.541ZINFOcrucible-agent (datafile): requesting running snapshot 106fca46-fef1-4b71-bb4b-7b85d6deea47-9e302fab-288d-4912-8d43-d6946fbfde66 state: Requested
1022024-07-10T20:09:22.541ZINFOcrucible-agent (worker): applying SMF actions for region 106fca46-fef1-4b71-bb4b-7b85d6deea47 running snapshot 9e302fab-288d-4912-8d43-d6946fbfde66 (state Requested)...
1032024-07-10T20:09:22.541ZINFOcrucible-agent (dropshot): request completed latency_us = 31010 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:62942 req_id = 3418ec64-59a0-4829-9a0c-1ad1f84df4ae response_code = 200 uri = /crucible/0/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47/snapshots/9e302fab-288d-4912-8d43-d6946fbfde66/run
1042024-07-10T20:09:22.544ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-106fca46-fef1-4b71-bb4b-7b85d6deea47-9e302fab-288d-4912-8d43-d6946fbfde66
1052024-07-10T20:09:22.547ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-106fca46-fef1-4b71-bb4b-7b85d6deea47-9e302fab-288d-4912-8d43-d6946fbfde66
1062024-07-10T20:09:22.557ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-106fca46-fef1-4b71-bb4b-7b85d6deea47-9e302fab-288d-4912-8d43-d6946fbfde66
1072024-07-10T20:09:22.557ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/106fca46-fef1-4b71-bb4b-7b85d6deea47/.zfs/snapshot/9e302fab-288d-4912-8d43-d6946fbfde66
1082024-07-10T20:09:22.557ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1092024-07-10T20:09:22.557ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
1102024-07-10T20:09:22.558ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
1112024-07-10T20:09:22.558ZINFOcrucible-agent (worker): commit
1122024-07-10T20:09:22.561ZINFOcrucible-agent (worker): ok!
1132024-07-10T20:09:22.565ZINFOcrucible-agent (worker): SMF ok!
1142024-07-10T20:09:22.565ZINFOcrucible-agent (datafile): region 106fca46-fef1-4b71-bb4b-7b85d6deea47 running snapshot 9e302fab-288d-4912-8d43-d6946fbfde66 state: Requested -> Created