Max level shown:
1 [ Jun 20 21:03:50 Enabled. ]
2 [ Jun 20 21:03:50 Rereading configuration. ]
3 [ Jun 20 21:03:51 Rereading configuration. ]
4 [ Jun 20 21:03:53 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_ac756826-7d23-4a48-8f48-471e91447b90/crucible -l [fd00:1122:3344:101::1a]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-06-20T21:03:54.058ZINFOcrucible-agent: dataset: "oxp_ac756826-7d23-4a48-8f48-471e91447b90/crucible"
72024-06-20T21:03:54.059ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::1a]:32345
82024-06-20T21:03:54.059ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-06-20T21:03:54.077ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-06-20T21:03:54.167ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::1a]:32345
112024-06-20T21:05:06.169ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::a]:43161
122024-06-20T21:05:06.204ZINFOcrucible-agent (datafile): region 47d0a5b7-051f-467c-b049-f73c0df12f57 state: Requested
132024-06-20T21:05:06.211ZINFOcrucible-agent (dropshot): request completed latency_us = 15385 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43161 req_id = c094d02c-ec10-458c-a90f-efdc4dec28dd response_code = 200 uri = /crucible/0/regions
142024-06-20T21:05:06.211ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152024-06-20T21:05:06.223ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_ac756826-7d23-4a48-8f48-471e91447b90/crucible/regions/47d0a5b7-051f-467c-b049-f73c0df12f57
162024-06-20T21:05:06.223ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_ac756826-7d23-4a48-8f48-471e91447b90/crucible/regions/47d0a5b7-051f-467c-b049-f73c0df12f57
172024-06-20T21:05:06.271ZINFOcrucible-agent (worker): creating region Region { id: RegionId("47d0a5b7-051f-467c-b049-f73c0df12f57"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/47d0a5b7-051f-467c-b049-f73c0df12f57" region = 47d0a5b7-051f-467c-b049-f73c0df12f57
182024-06-20T21:05:06.273ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "47d0a5b7-051f-467c-b049-f73c0df12f57" "--data" "/data/regions/47d0a5b7-051f-467c-b049-f73c0df12f57" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 47d0a5b7-051f-467c-b049-f73c0df12f57
192024-06-20T21:05:06.551ZINFOcrucible-agent (worker): region files created ok region = 47d0a5b7-051f-467c-b049-f73c0df12f57
202024-06-20T21:05:06.551ZINFOcrucible-agent (datafile): region 47d0a5b7-051f-467c-b049-f73c0df12f57 state: Requested -> Created
212024-06-20T21:05:06.551ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-06-20T21:05:06.551ZINFOcrucible-agent (dropshot): request completed latency_us = 88 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43161 req_id = 5e841ed4-7b7c-4825-b635-2bfde5b1deda response_code = 200 uri = /crucible/0/regions
232024-06-20T21:05:06.551ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-47d0a5b7-051f-467c-b049-f73c0df12f57
242024-06-20T21:05:06.557ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-47d0a5b7-051f-467c-b049-f73c0df12f57
252024-06-20T21:05:06.568ZINFOcrucible-agent (worker): creating config property group
262024-06-20T21:05:06.573ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-47d0a5b7-051f-467c-b049-f73c0df12f57
272024-06-20T21:05:06.581ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/47d0a5b7-051f-467c-b049-f73c0df12f57
282024-06-20T21:05:06.584ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-06-20T21:05:06.584ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
302024-06-20T21:05:06.584ZINFOcrucible-agent (worker): commit
312024-06-20T21:05:06.588ZINFOcrucible-agent (worker): ok!
322024-06-20T21:05:06.594ZINFOcrucible-agent (worker): SMF ok!
332024-06-20T21:10:19.031ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::a]:57540
342024-06-20T21:10:19.036ZINFOcrucible-agent (dropshot): request completed latency_us = 5142 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57540 req_id = db8a2f40-2b58-42e9-83cb-ca973da62c87 response_code = 200 uri = /crucible/0/regions/47d0a5b7-051f-467c-b049-f73c0df12f57
352024-06-20T21:10:19.036ZINFOcrucible-agent (datafile): path is "/data/regions/47d0a5b7-051f-467c-b049-f73c0df12f57"
362024-06-20T21:10:19.046ZINFOcrucible-agent (datafile): dataset is oxp_ac756826-7d23-4a48-8f48-471e91447b90/crucible/regions/47d0a5b7-051f-467c-b049-f73c0df12f57
372024-06-20T21:10:19.056ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_ac756826-7d23-4a48-8f48-471e91447b90/crucible/regions/47d0a5b7-051f-467c-b049-f73c0df12f57@1754a985-200c-4441-a679-cc9cc7979745
382024-06-20T21:10:19.056ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_ac756826-7d23-4a48-8f48-471e91447b90/crucible/regions/47d0a5b7-051f-467c-b049-f73c0df12f57", "1754a985-200c-4441-a679-cc9cc7979745"]
392024-06-20T21:10:19.070ZINFOcrucible-agent (dropshot): request completed latency_us = 33334 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57540 req_id = 03b91a85-89bf-452e-9408-ee5e1e3649e4 response_code = 200 uri = /crucible/0/regions/47d0a5b7-051f-467c-b049-f73c0df12f57/snapshots/1754a985-200c-4441-a679-cc9cc7979745
402024-06-20T21:10:19.070ZINFOcrucible-agent (datafile): path is "/data/regions/47d0a5b7-051f-467c-b049-f73c0df12f57"
412024-06-20T21:10:19.078ZINFOcrucible-agent (datafile): dataset is oxp_ac756826-7d23-4a48-8f48-471e91447b90/crucible/regions/47d0a5b7-051f-467c-b049-f73c0df12f57
422024-06-20T21:10:19.086ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_ac756826-7d23-4a48-8f48-471e91447b90/crucible/regions/47d0a5b7-051f-467c-b049-f73c0df12f57@1754a985-200c-4441-a679-cc9cc7979745
432024-06-20T21:10:19.086ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_ac756826-7d23-4a48-8f48-471e91447b90/crucible/regions/47d0a5b7-051f-467c-b049-f73c0df12f57", "1754a985-200c-4441-a679-cc9cc7979745"]
442024-06-20T21:10:19.095ZINFOcrucible-agent (datafile): requesting running snapshot 47d0a5b7-051f-467c-b049-f73c0df12f57-1754a985-200c-4441-a679-cc9cc7979745 state: Requested
452024-06-20T21:10:19.096ZINFOcrucible-agent (worker): applying SMF actions for region 47d0a5b7-051f-467c-b049-f73c0df12f57 running snapshot 1754a985-200c-4441-a679-cc9cc7979745 (state Requested)...
462024-06-20T21:10:19.096ZINFOcrucible-agent (dropshot): request completed latency_us = 25956 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57540 req_id = 0abe9059-7782-45b0-a86e-db94ab50acc2 response_code = 200 uri = /crucible/0/regions/47d0a5b7-051f-467c-b049-f73c0df12f57/snapshots/1754a985-200c-4441-a679-cc9cc7979745/run
472024-06-20T21:10:19.100ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/47d0a5b7-051f-467c-b049-f73c0df12f57
482024-06-20T21:10:19.100ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-06-20T21:10:19.100ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::1a
502024-06-20T21:10:19.100ZINFOcrucible-agent (worker): reconfig required
512024-06-20T21:10:19.100ZINFOcrucible-agent (worker): using existing config property group
522024-06-20T21:10:19.100ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-47d0a5b7-051f-467c-b049-f73c0df12f57
532024-06-20T21:10:19.101ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/47d0a5b7-051f-467c-b049-f73c0df12f57
542024-06-20T21:10:19.101ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-06-20T21:10:19.101ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
562024-06-20T21:10:19.101ZINFOcrucible-agent (worker): commit
572024-06-20T21:10:19.111ZINFOcrucible-agent (worker): ok!
582024-06-20T21:10:19.111ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-47d0a5b7-051f-467c-b049-f73c0df12f57-1754a985-200c-4441-a679-cc9cc7979745
592024-06-20T21:10:19.119ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-47d0a5b7-051f-467c-b049-f73c0df12f57-1754a985-200c-4441-a679-cc9cc7979745
602024-06-20T21:10:19.141ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-47d0a5b7-051f-467c-b049-f73c0df12f57-1754a985-200c-4441-a679-cc9cc7979745
612024-06-20T21:10:19.141ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/47d0a5b7-051f-467c-b049-f73c0df12f57/.zfs/snapshot/1754a985-200c-4441-a679-cc9cc7979745
622024-06-20T21:10:19.141ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
632024-06-20T21:10:19.141ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-06-20T21:10:19.141ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
652024-06-20T21:10:19.141ZINFOcrucible-agent (worker): commit
662024-06-20T21:10:19.147ZINFOcrucible-agent (worker): ok!
672024-06-20T21:10:19.154ZINFOcrucible-agent (worker): SMF ok!
682024-06-20T21:10:19.154ZINFOcrucible-agent (datafile): region 47d0a5b7-051f-467c-b049-f73c0df12f57 running snapshot 1754a985-200c-4441-a679-cc9cc7979745 state: Requested -> Created
692024-06-20T21:10:22.336ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::a]:51131
702024-06-20T21:10:22.336ZINFOcrucible-agent (datafile): region 90164b69-e614-45c0-a835-b11bbae4d96d state: Requested
712024-06-20T21:10:22.337ZINFOcrucible-agent (dropshot): request completed latency_us = 203 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51131 req_id = 9fe86927-a514-40dd-8779-f7390cc8b3c8 response_code = 200 uri = /crucible/0/regions
722024-06-20T21:10:22.337ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
732024-06-20T21:10:22.346ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_ac756826-7d23-4a48-8f48-471e91447b90/crucible/regions/90164b69-e614-45c0-a835-b11bbae4d96d
742024-06-20T21:10:22.346ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_ac756826-7d23-4a48-8f48-471e91447b90/crucible/regions/90164b69-e614-45c0-a835-b11bbae4d96d
752024-06-20T21:10:22.416ZINFOcrucible-agent (worker): creating region Region { id: RegionId("90164b69-e614-45c0-a835-b11bbae4d96d"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/90164b69-e614-45c0-a835-b11bbae4d96d" region = 90164b69-e614-45c0-a835-b11bbae4d96d
762024-06-20T21:10:22.416ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "90164b69-e614-45c0-a835-b11bbae4d96d" "--data" "/data/regions/90164b69-e614-45c0-a835-b11bbae4d96d" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 90164b69-e614-45c0-a835-b11bbae4d96d
772024-06-20T21:10:22.529ZINFOcrucible-agent (dropshot): request completed latency_us = 85 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51131 req_id = b3ed5a80-b51d-4c22-a1aa-6d72e8cdba01 response_code = 200 uri = /crucible/0/regions
782024-06-20T21:10:22.539ZINFOcrucible-agent (worker): region files created ok region = 90164b69-e614-45c0-a835-b11bbae4d96d
792024-06-20T21:10:22.539ZINFOcrucible-agent (datafile): region 90164b69-e614-45c0-a835-b11bbae4d96d state: Requested -> Created
802024-06-20T21:10:22.539ZINFOcrucible-agent (worker): applying SMF actions post create...
812024-06-20T21:10:22.540ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/47d0a5b7-051f-467c-b049-f73c0df12f57
822024-06-20T21:10:22.540ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
832024-06-20T21:10:22.540ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::1a
842024-06-20T21:10:22.540ZINFOcrucible-agent (worker): reconfig required
852024-06-20T21:10:22.540ZINFOcrucible-agent (worker): using existing config property group
862024-06-20T21:10:22.540ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-47d0a5b7-051f-467c-b049-f73c0df12f57
872024-06-20T21:10:22.541ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/47d0a5b7-051f-467c-b049-f73c0df12f57
882024-06-20T21:10:22.541ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
892024-06-20T21:10:22.541ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
902024-06-20T21:10:22.541ZINFOcrucible-agent (worker): commit
912024-06-20T21:10:22.548ZINFOcrucible-agent (worker): ok!
922024-06-20T21:10:22.548ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-90164b69-e614-45c0-a835-b11bbae4d96d
932024-06-20T21:10:22.558ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-90164b69-e614-45c0-a835-b11bbae4d96d
942024-06-20T21:10:22.578ZINFOcrucible-agent (worker): creating config property group
952024-06-20T21:10:22.584ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-90164b69-e614-45c0-a835-b11bbae4d96d
962024-06-20T21:10:22.584ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/90164b69-e614-45c0-a835-b11bbae4d96d
972024-06-20T21:10:22.584ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
982024-06-20T21:10:22.584ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
992024-06-20T21:10:22.584ZINFOcrucible-agent (worker): commit
1002024-06-20T21:10:22.591ZINFOcrucible-agent (worker): ok!
1012024-06-20T21:10:22.600ZINFOcrucible-agent (worker): SMF ok!
1022024-06-20T21:10:23.218ZINFOcrucible-agent (dropshot): request completed latency_us = 589 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51131 req_id = 61c2f6c3-c1bb-4228-b552-33f66fdd5da0 response_code = 200 uri = /crucible/0/regions
1032024-06-20T21:10:47.684ZINFOcrucible-agent (dropshot): request completed latency_us = 64 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51131 req_id = e8b5b6b6-edd5-4718-87a4-9c53565b2070 response_code = 200 uri = /crucible/0/regions/90164b69-e614-45c0-a835-b11bbae4d96d
1042024-06-20T21:10:47.687ZINFOcrucible-agent (datafile): path is "/data/regions/90164b69-e614-45c0-a835-b11bbae4d96d"
1052024-06-20T21:10:47.697ZINFOcrucible-agent (datafile): dataset is oxp_ac756826-7d23-4a48-8f48-471e91447b90/crucible/regions/90164b69-e614-45c0-a835-b11bbae4d96d
1062024-06-20T21:10:47.705ZINFOcrucible-agent (datafile): region 90164b69-e614-45c0-a835-b11bbae4d96d state: Created -> Tombstoned
1072024-06-20T21:10:47.706ZINFOcrucible-agent (worker): applying SMF actions before removal...
1082024-06-20T21:10:47.708ZINFOcrucible-agent (dropshot): request completed latency_us = 20846 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:51131 req_id = 2715ec44-490f-4898-bba2-53a430f7d0c4 response_code = 204 uri = /crucible/0/regions/90164b69-e614-45c0-a835-b11bbae4d96d
1092024-06-20T21:10:47.709ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-90164b69-e614-45c0-a835-b11bbae4d96d (instance states: (Some(Online), None))
1102024-06-20T21:10:47.710ZINFOcrucible-agent (dropshot): request completed latency_us = 31 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51131 req_id = 4a9c6287-0092-4290-949e-58fa36569a33 response_code = 200 uri = /crucible/0/regions/90164b69-e614-45c0-a835-b11bbae4d96d
1112024-06-20T21:10:47.713ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/47d0a5b7-051f-467c-b049-f73c0df12f57
1122024-06-20T21:10:47.714ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1132024-06-20T21:10:47.714ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::1a
1142024-06-20T21:10:47.714ZINFOcrucible-agent (worker): reconfig required
1152024-06-20T21:10:47.714ZINFOcrucible-agent (worker): using existing config property group
1162024-06-20T21:10:47.714ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-47d0a5b7-051f-467c-b049-f73c0df12f57
1172024-06-20T21:10:47.714ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/47d0a5b7-051f-467c-b049-f73c0df12f57
1182024-06-20T21:10:47.714ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1192024-06-20T21:10:47.714ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
1202024-06-20T21:10:47.714ZINFOcrucible-agent (worker): commit
1212024-06-20T21:10:47.721ZINFOcrucible-agent (worker): ok!
1222024-06-20T21:10:47.722ZINFOcrucible-agent (worker): SMF ok!
1232024-06-20T21:10:47.732ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_ac756826-7d23-4a48-8f48-471e91447b90/crucible/regions/90164b69-e614-45c0-a835-b11bbae4d96d" region = 90164b69-e614-45c0-a835-b11bbae4d96d
1242024-06-20T21:10:47.785ZINFOcrucible-agent (datafile): region 90164b69-e614-45c0-a835-b11bbae4d96d state: Tombstoned -> Destroyed
1252024-06-20T21:10:47.816ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51131 req_id = f18a222e-a522-4c68-88ab-e25122751568 response_code = 200 uri = /crucible/0/regions/90164b69-e614-45c0-a835-b11bbae4d96d