Max level shown:
1 [ Sep 3 23:51:12 Enabled. ]
2 [ Sep 3 23:51:12 Rereading configuration. ]
3 [ Sep 3 23:51:14 Rereading configuration. ]
4 [ Sep 3 23:51:15 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_7d26dc1e-4276-483d-aac3-2ccee06bc9a7/crucible -l [fd00:1122:3344:101::1a]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-09-03T23:51:15.796ZINFOcrucible-agent: dataset: "oxp_7d26dc1e-4276-483d-aac3-2ccee06bc9a7/crucible"
72024-09-03T23:51:15.796ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::1a]:32345
82024-09-03T23:51:15.796ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-09-03T23:51:15.817ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-09-03T23:51:15.956ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::1a]:32345
112024-09-03T23:51:47.126ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::a]:62255
122024-09-03T23:51:47.150ZINFOcrucible-agent (datafile): region df5fb09f-e3c5-44e6-a284-4a4b372d2fad state: Requested
132024-09-03T23:51:47.152ZINFOcrucible-agent (dropshot): request completed latency_us = 17310 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:62255 req_id = 31ebf0bd-1d41-436d-9d86-9159fdc2f549 response_code = 200 uri = /crucible/0/regions
142024-09-03T23:51:47.154ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152024-09-03T23:51:47.164ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_7d26dc1e-4276-483d-aac3-2ccee06bc9a7/crucible/regions/df5fb09f-e3c5-44e6-a284-4a4b372d2fad
162024-09-03T23:51:47.164ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_7d26dc1e-4276-483d-aac3-2ccee06bc9a7/crucible/regions/df5fb09f-e3c5-44e6-a284-4a4b372d2fad
172024-09-03T23:51:47.230ZINFOcrucible-agent (worker): creating region Region { id: RegionId("df5fb09f-e3c5-44e6-a284-4a4b372d2fad"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/df5fb09f-e3c5-44e6-a284-4a4b372d2fad" region = df5fb09f-e3c5-44e6-a284-4a4b372d2fad
182024-09-03T23:51:47.232ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "df5fb09f-e3c5-44e6-a284-4a4b372d2fad" "--data" "/data/regions/df5fb09f-e3c5-44e6-a284-4a4b372d2fad" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = df5fb09f-e3c5-44e6-a284-4a4b372d2fad
192024-09-03T23:51:47.461ZINFOcrucible-agent (worker): region files created ok region = df5fb09f-e3c5-44e6-a284-4a4b372d2fad
202024-09-03T23:51:47.461ZINFOcrucible-agent (datafile): region df5fb09f-e3c5-44e6-a284-4a4b372d2fad state: Requested -> Created
212024-09-03T23:51:47.461ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-09-03T23:51:47.462ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-df5fb09f-e3c5-44e6-a284-4a4b372d2fad
232024-09-03T23:51:47.465ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-df5fb09f-e3c5-44e6-a284-4a4b372d2fad
242024-09-03T23:51:47.477ZINFOcrucible-agent (worker): creating config property group
252024-09-03T23:51:47.480ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-df5fb09f-e3c5-44e6-a284-4a4b372d2fad
262024-09-03T23:51:47.489ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/df5fb09f-e3c5-44e6-a284-4a4b372d2fad
272024-09-03T23:51:47.491ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-09-03T23:51:47.491ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
292024-09-03T23:51:47.491ZINFOcrucible-agent (worker): commit
302024-09-03T23:51:47.497ZINFOcrucible-agent (worker): ok!
312024-09-03T23:51:47.505ZINFOcrucible-agent (worker): SMF ok!
322024-09-03T23:51:47.508ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:62255 req_id = d3969526-5ca6-4639-8710-5406731c6416 response_code = 200 uri = /crucible/0/regions
332024-09-03T23:51:48.631ZINFOcrucible-agent (dropshot): request completed latency_us = 83 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:62255 req_id = efc48a4a-773f-4271-aad3-8b3b1cdd1a70 response_code = 200 uri = /crucible/0/regions/df5fb09f-e3c5-44e6-a284-4a4b372d2fad
342024-09-03T23:51:48.634ZINFOcrucible-agent (datafile): path is "/data/regions/df5fb09f-e3c5-44e6-a284-4a4b372d2fad"
352024-09-03T23:51:48.644ZINFOcrucible-agent (datafile): dataset is oxp_7d26dc1e-4276-483d-aac3-2ccee06bc9a7/crucible/regions/df5fb09f-e3c5-44e6-a284-4a4b372d2fad
362024-09-03T23:51:48.653ZINFOcrucible-agent (datafile): region df5fb09f-e3c5-44e6-a284-4a4b372d2fad state: Created -> Tombstoned
372024-09-03T23:51:48.653ZINFOcrucible-agent (worker): applying SMF actions before removal...
382024-09-03T23:51:48.653ZINFOcrucible-agent (dropshot): request completed latency_us = 19044 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:62255 req_id = 01b0398c-e027-44c5-8d9d-8b81bcda4b34 response_code = 204 uri = /crucible/0/regions/df5fb09f-e3c5-44e6-a284-4a4b372d2fad
392024-09-03T23:51:48.655ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-df5fb09f-e3c5-44e6-a284-4a4b372d2fad (instance states: (Some(Online), None))
402024-09-03T23:51:48.657ZINFOcrucible-agent (dropshot): request completed latency_us = 29 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:62255 req_id = eb2a2fd0-c2b7-46e4-9b1c-098d244cfaf9 response_code = 200 uri = /crucible/0/regions/df5fb09f-e3c5-44e6-a284-4a4b372d2fad
412024-09-03T23:51:48.659ZINFOcrucible-agent (worker): SMF ok!
422024-09-03T23:51:48.669ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_7d26dc1e-4276-483d-aac3-2ccee06bc9a7/crucible/regions/df5fb09f-e3c5-44e6-a284-4a4b372d2fad" region = df5fb09f-e3c5-44e6-a284-4a4b372d2fad
432024-09-03T23:51:48.725ZINFOcrucible-agent (datafile): region df5fb09f-e3c5-44e6-a284-4a4b372d2fad state: Tombstoned -> Destroyed
442024-09-03T23:51:48.774ZINFOcrucible-agent (dropshot): request completed latency_us = 63 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:62255 req_id = a6933dfb-87ee-4370-8558-c63a256473de response_code = 200 uri = /crucible/0/regions/df5fb09f-e3c5-44e6-a284-4a4b372d2fad
452024-09-03T23:57:01.985ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::a]:52284
462024-09-03T23:57:01.985ZINFOcrucible-agent (datafile): region 0391e41a-a2ac-44a6-aa32-74fe6bedd913 state: Requested
472024-09-03T23:57:01.985ZINFOcrucible-agent (dropshot): request completed latency_us = 733 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52284 req_id = 32df4ecb-7ccd-41fb-8427-17829473ac8d response_code = 200 uri = /crucible/0/regions
482024-09-03T23:57:01.985ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
492024-09-03T23:57:01.997ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_7d26dc1e-4276-483d-aac3-2ccee06bc9a7/crucible/regions/0391e41a-a2ac-44a6-aa32-74fe6bedd913
502024-09-03T23:57:01.997ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_7d26dc1e-4276-483d-aac3-2ccee06bc9a7/crucible/regions/0391e41a-a2ac-44a6-aa32-74fe6bedd913
512024-09-03T23:57:02.052ZINFOcrucible-agent (worker): creating region Region { id: RegionId("0391e41a-a2ac-44a6-aa32-74fe6bedd913"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/0391e41a-a2ac-44a6-aa32-74fe6bedd913" region = 0391e41a-a2ac-44a6-aa32-74fe6bedd913
522024-09-03T23:57:02.054ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "0391e41a-a2ac-44a6-aa32-74fe6bedd913" "--data" "/data/regions/0391e41a-a2ac-44a6-aa32-74fe6bedd913" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 0391e41a-a2ac-44a6-aa32-74fe6bedd913
532024-09-03T23:57:02.126ZINFOcrucible-agent (worker): region files created ok region = 0391e41a-a2ac-44a6-aa32-74fe6bedd913
542024-09-03T23:57:02.126ZINFOcrucible-agent (datafile): region 0391e41a-a2ac-44a6-aa32-74fe6bedd913 state: Requested -> Created
552024-09-03T23:57:02.127ZINFOcrucible-agent (worker): applying SMF actions post create...
562024-09-03T23:57:02.128ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-0391e41a-a2ac-44a6-aa32-74fe6bedd913
572024-09-03T23:57:02.134ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-0391e41a-a2ac-44a6-aa32-74fe6bedd913
582024-09-03T23:57:02.147ZINFOcrucible-agent (worker): creating config property group
592024-09-03T23:57:02.151ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0391e41a-a2ac-44a6-aa32-74fe6bedd913
602024-09-03T23:57:02.152ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0391e41a-a2ac-44a6-aa32-74fe6bedd913
612024-09-03T23:57:02.152ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
622024-09-03T23:57:02.152ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
632024-09-03T23:57:02.152ZINFOcrucible-agent (worker): commit
642024-09-03T23:57:02.155ZINFOcrucible-agent (worker): ok!
652024-09-03T23:57:02.161ZINFOcrucible-agent (worker): SMF ok!
662024-09-03T23:57:02.332ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52284 req_id = c12123bf-0e38-4ef7-ba2a-b12c3443f4be response_code = 200 uri = /crucible/0/regions
672024-09-03T23:57:27.610ZINFOcrucible-agent (dropshot): request completed latency_us = 50 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:52284 req_id = 0eb30ab1-ff0d-4700-bd2d-d62ab3cddf9c response_code = 200 uri = /crucible/0/regions/0391e41a-a2ac-44a6-aa32-74fe6bedd913
682024-09-03T23:57:27.613ZINFOcrucible-agent (datafile): path is "/data/regions/0391e41a-a2ac-44a6-aa32-74fe6bedd913"
692024-09-03T23:57:27.623ZINFOcrucible-agent (datafile): dataset is oxp_7d26dc1e-4276-483d-aac3-2ccee06bc9a7/crucible/regions/0391e41a-a2ac-44a6-aa32-74fe6bedd913
702024-09-03T23:57:27.631ZINFOcrucible-agent (datafile): region 0391e41a-a2ac-44a6-aa32-74fe6bedd913 state: Created -> Tombstoned
712024-09-03T23:57:27.632ZINFOcrucible-agent (worker): applying SMF actions before removal...
722024-09-03T23:57:27.632ZINFOcrucible-agent (dropshot): request completed latency_us = 18838 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:52284 req_id = 3f274e85-db1d-4759-867f-6d51ef0956ed response_code = 204 uri = /crucible/0/regions/0391e41a-a2ac-44a6-aa32-74fe6bedd913
732024-09-03T23:57:27.633ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-0391e41a-a2ac-44a6-aa32-74fe6bedd913 (instance states: (Some(Online), None))
742024-09-03T23:57:27.635ZINFOcrucible-agent (dropshot): request completed latency_us = 24 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:52284 req_id = b4ec64d2-0a88-43b4-8cd4-efa89068f401 response_code = 200 uri = /crucible/0/regions/0391e41a-a2ac-44a6-aa32-74fe6bedd913
752024-09-03T23:57:27.637ZINFOcrucible-agent (worker): SMF ok!
762024-09-03T23:57:27.647ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_7d26dc1e-4276-483d-aac3-2ccee06bc9a7/crucible/regions/0391e41a-a2ac-44a6-aa32-74fe6bedd913" region = 0391e41a-a2ac-44a6-aa32-74fe6bedd913
772024-09-03T23:57:27.695ZINFOcrucible-agent (datafile): region 0391e41a-a2ac-44a6-aa32-74fe6bedd913 state: Tombstoned -> Destroyed
782024-09-03T23:57:27.779ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:52284 req_id = abf4314c-b241-4370-b566-778a0e0b1be7 response_code = 200 uri = /crucible/0/regions/0391e41a-a2ac-44a6-aa32-74fe6bedd913