Max level shown:
1 [ Jul 25 05:48:05 Enabled. ]
2 [ Jul 25 05:48:05 Rereading configuration. ]
3 [ Jul 25 05:48:10 Rereading configuration. ]
4 [ Jul 25 05:48:13 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible -l [fd00:1122:3344:101::15]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-07-25T05:48:13.737ZINFOcrucible-agent: dataset: "oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible"
72024-07-25T05:48:13.749ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::15]:32345
82024-07-25T05:48:13.749ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-07-25T05:48:13.777ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-07-25T05:48:13.937ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::15]:32345
112024-07-25T05:48:35.639ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::c]:51502
122024-07-25T05:48:35.656ZINFOcrucible-agent (datafile): region 795a06de-b6d7-4454-b865-904ccae0f9e2 state: Requested
132024-07-25T05:48:35.664ZINFOcrucible-agent (dropshot): request completed latency_us = 14453 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:51502 req_id = 1e5f7de9-dd49-4324-9cbc-27277af3d052 response_code = 200 uri = /crucible/0/regions
142024-07-25T05:48:35.666ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152024-07-25T05:48:35.675ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible/regions/795a06de-b6d7-4454-b865-904ccae0f9e2
162024-07-25T05:48:35.675ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible/regions/795a06de-b6d7-4454-b865-904ccae0f9e2
172024-07-25T05:48:35.730ZINFOcrucible-agent (worker): creating region Region { id: RegionId("795a06de-b6d7-4454-b865-904ccae0f9e2"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/795a06de-b6d7-4454-b865-904ccae0f9e2" region = 795a06de-b6d7-4454-b865-904ccae0f9e2
182024-07-25T05:48:35.732ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "795a06de-b6d7-4454-b865-904ccae0f9e2" "--data" "/data/regions/795a06de-b6d7-4454-b865-904ccae0f9e2" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 795a06de-b6d7-4454-b865-904ccae0f9e2
192024-07-25T05:48:35.848ZINFOcrucible-agent (worker): region files created ok region = 795a06de-b6d7-4454-b865-904ccae0f9e2
202024-07-25T05:48:35.848ZINFOcrucible-agent (datafile): region 795a06de-b6d7-4454-b865-904ccae0f9e2 state: Requested -> Created
212024-07-25T05:48:35.848ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-07-25T05:48:35.849ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-795a06de-b6d7-4454-b865-904ccae0f9e2
232024-07-25T05:48:35.854ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-795a06de-b6d7-4454-b865-904ccae0f9e2
242024-07-25T05:48:35.855ZINFOcrucible-agent (dropshot): request completed latency_us = 68 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:51502 req_id = 2bcd3a9b-4942-4f65-bfe8-e8288023195e response_code = 200 uri = /crucible/0/regions
252024-07-25T05:48:35.874ZINFOcrucible-agent (worker): creating config property group
262024-07-25T05:48:35.883ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-795a06de-b6d7-4454-b865-904ccae0f9e2
272024-07-25T05:48:35.884ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/795a06de-b6d7-4454-b865-904ccae0f9e2
282024-07-25T05:48:35.888ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-07-25T05:48:35.888ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
302024-07-25T05:48:35.888ZINFOcrucible-agent (worker): commit
312024-07-25T05:48:35.895ZINFOcrucible-agent (worker): ok!
322024-07-25T05:48:35.899ZINFOcrucible-agent (worker): SMF ok!
332024-07-25T05:48:36.608ZINFOcrucible-agent (dropshot): request completed latency_us = 102 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:51502 req_id = b8a85ea1-e122-401b-a28d-66ebe0b7166c response_code = 200 uri = /crucible/0/regions/795a06de-b6d7-4454-b865-904ccae0f9e2
342024-07-25T05:48:36.612ZINFOcrucible-agent (datafile): path is "/data/regions/795a06de-b6d7-4454-b865-904ccae0f9e2"
352024-07-25T05:48:36.620ZINFOcrucible-agent (datafile): dataset is oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible/regions/795a06de-b6d7-4454-b865-904ccae0f9e2
362024-07-25T05:48:36.628ZINFOcrucible-agent (datafile): region 795a06de-b6d7-4454-b865-904ccae0f9e2 state: Created -> Tombstoned
372024-07-25T05:48:36.629ZINFOcrucible-agent (worker): applying SMF actions before removal...
382024-07-25T05:48:36.629ZINFOcrucible-agent (dropshot): request completed latency_us = 18002 local_addr = [fd00:1122:3344:101::15]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:51502 req_id = 6a7c9037-b79d-40f5-ab11-710414ed3a9b response_code = 204 uri = /crucible/0/regions/795a06de-b6d7-4454-b865-904ccae0f9e2
392024-07-25T05:48:36.630ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-795a06de-b6d7-4454-b865-904ccae0f9e2 (instance states: (Some(Online), None))
402024-07-25T05:48:36.632ZINFOcrucible-agent (dropshot): request completed latency_us = 21 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:51502 req_id = 84ad21a9-deec-46a4-89c3-eb632cceab5b response_code = 200 uri = /crucible/0/regions/795a06de-b6d7-4454-b865-904ccae0f9e2
412024-07-25T05:48:36.634ZINFOcrucible-agent (worker): SMF ok!
422024-07-25T05:48:36.643ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible/regions/795a06de-b6d7-4454-b865-904ccae0f9e2" region = 795a06de-b6d7-4454-b865-904ccae0f9e2
432024-07-25T05:48:36.696ZINFOcrucible-agent (datafile): region 795a06de-b6d7-4454-b865-904ccae0f9e2 state: Tombstoned -> Destroyed
442024-07-25T05:48:36.735ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:51502 req_id = 9f2b5fa3-d39d-4a71-ba6d-c9b4179b10d8 response_code = 200 uri = /crucible/0/regions/795a06de-b6d7-4454-b865-904ccae0f9e2
452024-07-25T05:49:26.252ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:51169
462024-07-25T05:49:26.253ZINFOcrucible-agent (datafile): region 6744bea5-2d71-4283-829d-903a2c16b105 state: Requested
472024-07-25T05:49:26.256ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
482024-07-25T05:49:26.256ZINFOcrucible-agent (dropshot): request completed latency_us = 3040 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51169 req_id = afaa1017-21d3-4e69-8f10-45eb3af60fd6 response_code = 200 uri = /crucible/0/regions
492024-07-25T05:49:26.264ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible/regions/6744bea5-2d71-4283-829d-903a2c16b105
502024-07-25T05:49:26.264ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible/regions/6744bea5-2d71-4283-829d-903a2c16b105
512024-07-25T05:49:26.314ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6744bea5-2d71-4283-829d-903a2c16b105"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/6744bea5-2d71-4283-829d-903a2c16b105" region = 6744bea5-2d71-4283-829d-903a2c16b105
522024-07-25T05:49:26.315ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "6744bea5-2d71-4283-829d-903a2c16b105" "--data" "/data/regions/6744bea5-2d71-4283-829d-903a2c16b105" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 6744bea5-2d71-4283-829d-903a2c16b105
532024-07-25T05:49:26.382ZINFOcrucible-agent (worker): region files created ok region = 6744bea5-2d71-4283-829d-903a2c16b105
542024-07-25T05:49:26.382ZINFOcrucible-agent (datafile): region 6744bea5-2d71-4283-829d-903a2c16b105 state: Requested -> Created
552024-07-25T05:49:26.382ZINFOcrucible-agent (worker): applying SMF actions post create...
562024-07-25T05:49:26.384ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6744bea5-2d71-4283-829d-903a2c16b105
572024-07-25T05:49:26.387ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6744bea5-2d71-4283-829d-903a2c16b105
582024-07-25T05:49:26.395ZINFOcrucible-agent (worker): creating config property group
592024-07-25T05:49:26.398ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6744bea5-2d71-4283-829d-903a2c16b105
602024-07-25T05:49:26.398ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6744bea5-2d71-4283-829d-903a2c16b105
612024-07-25T05:49:26.398ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
622024-07-25T05:49:26.398ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
632024-07-25T05:49:26.398ZINFOcrucible-agent (worker): commit
642024-07-25T05:49:26.402ZINFOcrucible-agent (worker): ok!
652024-07-25T05:49:26.406ZINFOcrucible-agent (worker): SMF ok!
662024-07-25T05:49:26.604ZINFOcrucible-agent (dropshot): request completed latency_us = 64 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51169 req_id = 5a3b81dc-ee7b-4bbe-b55b-734ed7e473a8 response_code = 200 uri = /crucible/0/regions
672024-07-25T05:54:52.875ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:37490
682024-07-25T05:54:52.875ZINFOcrucible-agent (datafile): region 347d8b2d-3b3e-4674-9eb8-3ae8ec2301c2 state: Requested
692024-07-25T05:54:52.878ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
702024-07-25T05:54:52.878ZINFOcrucible-agent (dropshot): request completed latency_us = 3075 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:37490 req_id = 7ae65d04-ab32-4e1d-88d6-7de30b797928 response_code = 200 uri = /crucible/0/regions
712024-07-25T05:54:52.887ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible/regions/347d8b2d-3b3e-4674-9eb8-3ae8ec2301c2
722024-07-25T05:54:52.887ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible/regions/347d8b2d-3b3e-4674-9eb8-3ae8ec2301c2
732024-07-25T05:54:52.994ZINFOcrucible-agent (worker): creating region Region { id: RegionId("347d8b2d-3b3e-4674-9eb8-3ae8ec2301c2"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/347d8b2d-3b3e-4674-9eb8-3ae8ec2301c2" region = 347d8b2d-3b3e-4674-9eb8-3ae8ec2301c2
742024-07-25T05:54:52.994ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "347d8b2d-3b3e-4674-9eb8-3ae8ec2301c2" "--data" "/data/regions/347d8b2d-3b3e-4674-9eb8-3ae8ec2301c2" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 347d8b2d-3b3e-4674-9eb8-3ae8ec2301c2
752024-07-25T05:54:53.195ZINFOcrucible-agent (dropshot): request completed latency_us = 86 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:37490 req_id = 09755dbc-7818-4f4f-9f62-a8ce78adf59a response_code = 200 uri = /crucible/0/regions
762024-07-25T05:54:53.244ZINFOcrucible-agent (worker): region files created ok region = 347d8b2d-3b3e-4674-9eb8-3ae8ec2301c2
772024-07-25T05:54:53.244ZINFOcrucible-agent (datafile): region 347d8b2d-3b3e-4674-9eb8-3ae8ec2301c2 state: Requested -> Created
782024-07-25T05:54:53.244ZINFOcrucible-agent (worker): applying SMF actions post create...
792024-07-25T05:54:53.245ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-347d8b2d-3b3e-4674-9eb8-3ae8ec2301c2
802024-07-25T05:54:53.286ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-347d8b2d-3b3e-4674-9eb8-3ae8ec2301c2
812024-07-25T05:54:53.332ZINFOcrucible-agent (worker): creating config property group
822024-07-25T05:54:53.344ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-347d8b2d-3b3e-4674-9eb8-3ae8ec2301c2
832024-07-25T05:54:53.344ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/347d8b2d-3b3e-4674-9eb8-3ae8ec2301c2
842024-07-25T05:54:53.344ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
852024-07-25T05:54:53.344ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
862024-07-25T05:54:53.344ZINFOcrucible-agent (worker): commit
872024-07-25T05:54:53.353ZINFOcrucible-agent (worker): ok!
882024-07-25T05:54:53.369ZINFOcrucible-agent (worker): SMF ok!
892024-07-25T05:54:53.753ZINFOcrucible-agent (dropshot): request completed latency_us = 619 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:37490 req_id = ba3dad4b-9d65-4243-a119-5baf578a1807 response_code = 200 uri = /crucible/0/regions
902024-07-25T05:54:54.753ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:45472
912024-07-25T05:54:54.753ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:45472 req_id = f36782fb-9930-43d3-9099-e1dc8f0c1c5c response_code = 200 uri = /crucible/0/regions/6744bea5-2d71-4283-829d-903a2c16b105
922024-07-25T05:54:54.753ZINFOcrucible-agent (datafile): path is "/data/regions/6744bea5-2d71-4283-829d-903a2c16b105"
932024-07-25T05:54:54.763ZINFOcrucible-agent (datafile): dataset is oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible/regions/6744bea5-2d71-4283-829d-903a2c16b105
942024-07-25T05:54:54.774ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible/regions/6744bea5-2d71-4283-829d-903a2c16b105@02c17e09-5830-4ee8-a416-a66468634fe2
952024-07-25T05:54:54.774ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible/regions/6744bea5-2d71-4283-829d-903a2c16b105", "02c17e09-5830-4ee8-a416-a66468634fe2"]
962024-07-25T05:54:54.788ZINFOcrucible-agent (dropshot): request completed latency_us = 35261 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:45472 req_id = c191beff-5616-4ff9-88e2-09c07fdc1322 response_code = 200 uri = /crucible/0/regions/6744bea5-2d71-4283-829d-903a2c16b105/snapshots/02c17e09-5830-4ee8-a416-a66468634fe2
972024-07-25T05:54:54.790ZINFOcrucible-agent (datafile): path is "/data/regions/6744bea5-2d71-4283-829d-903a2c16b105"
982024-07-25T05:54:54.801ZINFOcrucible-agent (datafile): dataset is oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible/regions/6744bea5-2d71-4283-829d-903a2c16b105
992024-07-25T05:54:54.811ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible/regions/6744bea5-2d71-4283-829d-903a2c16b105@02c17e09-5830-4ee8-a416-a66468634fe2
1002024-07-25T05:54:54.811ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible/regions/6744bea5-2d71-4283-829d-903a2c16b105", "02c17e09-5830-4ee8-a416-a66468634fe2"]
1012024-07-25T05:54:54.824ZINFOcrucible-agent (datafile): requesting running snapshot 6744bea5-2d71-4283-829d-903a2c16b105-02c17e09-5830-4ee8-a416-a66468634fe2 state: Requested
1022024-07-25T05:54:54.824ZINFOcrucible-agent (worker): applying SMF actions for region 6744bea5-2d71-4283-829d-903a2c16b105 running snapshot 02c17e09-5830-4ee8-a416-a66468634fe2 (state Requested)...
1032024-07-25T05:54:54.824ZINFOcrucible-agent (dropshot): request completed latency_us = 35349 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:45472 req_id = 754544c0-5a30-433b-a9f3-28907bad9393 response_code = 200 uri = /crucible/0/regions/6744bea5-2d71-4283-829d-903a2c16b105/snapshots/02c17e09-5830-4ee8-a416-a66468634fe2/run
1042024-07-25T05:54:54.827ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-6744bea5-2d71-4283-829d-903a2c16b105-02c17e09-5830-4ee8-a416-a66468634fe2
1052024-07-25T05:54:54.854ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-6744bea5-2d71-4283-829d-903a2c16b105-02c17e09-5830-4ee8-a416-a66468634fe2
1062024-07-25T05:54:54.889ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-6744bea5-2d71-4283-829d-903a2c16b105-02c17e09-5830-4ee8-a416-a66468634fe2
1072024-07-25T05:54:54.889ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6744bea5-2d71-4283-829d-903a2c16b105/.zfs/snapshot/02c17e09-5830-4ee8-a416-a66468634fe2
1082024-07-25T05:54:54.889ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1092024-07-25T05:54:54.889ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
1102024-07-25T05:54:54.889ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
1112024-07-25T05:54:54.889ZINFOcrucible-agent (worker): commit
1122024-07-25T05:54:54.902ZINFOcrucible-agent (worker): ok!
1132024-07-25T05:54:54.908ZINFOcrucible-agent (worker): SMF ok!
1142024-07-25T05:54:54.908ZINFOcrucible-agent (datafile): region 6744bea5-2d71-4283-829d-903a2c16b105 running snapshot 02c17e09-5830-4ee8-a416-a66468634fe2 state: Requested -> Created