Max level shown:
1 [ Jul 25 05:48:05 Enabled. ]
2 [ Jul 25 05:48:05 Rereading configuration. ]
3 [ Jul 25 05:48:10 Rereading configuration. ]
4 [ Jul 25 05:48:14 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_e43689a1-378e-4be8-9661-3bc522647c6d/crucible -l [fd00:1122:3344:101::1a]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-07-25T05:48:14.076ZINFOcrucible-agent: dataset: "oxp_e43689a1-378e-4be8-9661-3bc522647c6d/crucible"
72024-07-25T05:48:14.081ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::1a]:32345
82024-07-25T05:48:14.081ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-07-25T05:48:14.099ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-07-25T05:48:14.200ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::1a]:32345
112024-07-25T05:48:35.638ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::c]:47614
122024-07-25T05:48:35.645ZINFOcrucible-agent (datafile): region a347d57c-8edb-4f3e-a638-9ecfc14d4d23 state: Requested
132024-07-25T05:48:35.645ZINFOcrucible-agent (dropshot): request completed latency_us = 2479 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:47614 req_id = 21acfa57-4abe-4b07-b466-f457e4f28827 response_code = 200 uri = /crucible/0/regions
142024-07-25T05:48:35.646ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152024-07-25T05:48:35.656ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_e43689a1-378e-4be8-9661-3bc522647c6d/crucible/regions/a347d57c-8edb-4f3e-a638-9ecfc14d4d23
162024-07-25T05:48:35.656ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_e43689a1-378e-4be8-9661-3bc522647c6d/crucible/regions/a347d57c-8edb-4f3e-a638-9ecfc14d4d23
172024-07-25T05:48:35.712ZINFOcrucible-agent (worker): creating region Region { id: RegionId("a347d57c-8edb-4f3e-a638-9ecfc14d4d23"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/a347d57c-8edb-4f3e-a638-9ecfc14d4d23" region = a347d57c-8edb-4f3e-a638-9ecfc14d4d23
182024-07-25T05:48:35.714ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "a347d57c-8edb-4f3e-a638-9ecfc14d4d23" "--data" "/data/regions/a347d57c-8edb-4f3e-a638-9ecfc14d4d23" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = a347d57c-8edb-4f3e-a638-9ecfc14d4d23
192024-07-25T05:48:35.849ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:47614 req_id = b00a8416-d06e-4b6c-8f97-d2c6b959a799 response_code = 200 uri = /crucible/0/regions
202024-07-25T05:48:35.903ZINFOcrucible-agent (worker): region files created ok region = a347d57c-8edb-4f3e-a638-9ecfc14d4d23
212024-07-25T05:48:35.903ZINFOcrucible-agent (datafile): region a347d57c-8edb-4f3e-a638-9ecfc14d4d23 state: Requested -> Created
222024-07-25T05:48:35.903ZINFOcrucible-agent (worker): applying SMF actions post create...
232024-07-25T05:48:35.904ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-a347d57c-8edb-4f3e-a638-9ecfc14d4d23
242024-07-25T05:48:35.909ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-a347d57c-8edb-4f3e-a638-9ecfc14d4d23
252024-07-25T05:48:35.922ZINFOcrucible-agent (worker): creating config property group
262024-07-25T05:48:35.925ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-a347d57c-8edb-4f3e-a638-9ecfc14d4d23
272024-07-25T05:48:35.926ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a347d57c-8edb-4f3e-a638-9ecfc14d4d23
282024-07-25T05:48:35.930ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-07-25T05:48:35.930ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
302024-07-25T05:48:35.930ZINFOcrucible-agent (worker): commit
312024-07-25T05:48:35.937ZINFOcrucible-agent (worker): ok!
322024-07-25T05:48:35.944ZINFOcrucible-agent (worker): SMF ok!
332024-07-25T05:48:36.265ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:47614 req_id = e9f08ac1-e14c-4b9c-b607-fc53f6f8a883 response_code = 200 uri = /crucible/0/regions
342024-07-25T05:48:36.490ZINFOcrucible-agent (dropshot): request completed latency_us = 99 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:47614 req_id = afbf9c60-b19e-40ba-91b3-0273cddc4e3d response_code = 200 uri = /crucible/0/regions/a347d57c-8edb-4f3e-a638-9ecfc14d4d23
352024-07-25T05:48:36.493ZINFOcrucible-agent (datafile): path is "/data/regions/a347d57c-8edb-4f3e-a638-9ecfc14d4d23"
362024-07-25T05:48:36.502ZINFOcrucible-agent (datafile): dataset is oxp_e43689a1-378e-4be8-9661-3bc522647c6d/crucible/regions/a347d57c-8edb-4f3e-a638-9ecfc14d4d23
372024-07-25T05:48:36.511ZINFOcrucible-agent (datafile): region a347d57c-8edb-4f3e-a638-9ecfc14d4d23 state: Created -> Tombstoned
382024-07-25T05:48:36.511ZINFOcrucible-agent (worker): applying SMF actions before removal...
392024-07-25T05:48:36.511ZINFOcrucible-agent (dropshot): request completed latency_us = 18591 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:47614 req_id = f452a801-f573-49b3-b68d-5da6461506c9 response_code = 204 uri = /crucible/0/regions/a347d57c-8edb-4f3e-a638-9ecfc14d4d23
402024-07-25T05:48:36.512ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-a347d57c-8edb-4f3e-a638-9ecfc14d4d23 (instance states: (Some(Online), None))
412024-07-25T05:48:36.513ZINFOcrucible-agent (dropshot): request completed latency_us = 29 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:47614 req_id = 2eee7668-685c-4bbd-9541-bf0fb295fa8b response_code = 200 uri = /crucible/0/regions/a347d57c-8edb-4f3e-a638-9ecfc14d4d23
422024-07-25T05:48:36.516ZINFOcrucible-agent (worker): SMF ok!
432024-07-25T05:48:36.526ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_e43689a1-378e-4be8-9661-3bc522647c6d/crucible/regions/a347d57c-8edb-4f3e-a638-9ecfc14d4d23" region = a347d57c-8edb-4f3e-a638-9ecfc14d4d23
442024-07-25T05:48:36.577ZINFOcrucible-agent (datafile): region a347d57c-8edb-4f3e-a638-9ecfc14d4d23 state: Tombstoned -> Destroyed
452024-07-25T05:48:36.591ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:47614 req_id = fef600aa-ee91-4148-ab54-670d36256055 response_code = 200 uri = /crucible/0/regions/a347d57c-8edb-4f3e-a638-9ecfc14d4d23
462024-07-25T05:54:52.874ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::a]:38701
472024-07-25T05:54:52.875ZINFOcrucible-agent (datafile): region a035f0b9-ad4e-4c2d-a88e-dc8d80cc8f08 state: Requested
482024-07-25T05:54:52.878ZINFOcrucible-agent (dropshot): request completed latency_us = 3306 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38701 req_id = e38329e7-2b85-4387-85c2-f11751541fe5 response_code = 200 uri = /crucible/0/regions
492024-07-25T05:54:52.878ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
502024-07-25T05:54:52.890ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_e43689a1-378e-4be8-9661-3bc522647c6d/crucible/regions/a035f0b9-ad4e-4c2d-a88e-dc8d80cc8f08
512024-07-25T05:54:52.890ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_e43689a1-378e-4be8-9661-3bc522647c6d/crucible/regions/a035f0b9-ad4e-4c2d-a88e-dc8d80cc8f08
522024-07-25T05:54:52.975ZINFOcrucible-agent (worker): creating region Region { id: RegionId("a035f0b9-ad4e-4c2d-a88e-dc8d80cc8f08"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/a035f0b9-ad4e-4c2d-a88e-dc8d80cc8f08" region = a035f0b9-ad4e-4c2d-a88e-dc8d80cc8f08
532024-07-25T05:54:52.977ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "a035f0b9-ad4e-4c2d-a88e-dc8d80cc8f08" "--data" "/data/regions/a035f0b9-ad4e-4c2d-a88e-dc8d80cc8f08" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = a035f0b9-ad4e-4c2d-a88e-dc8d80cc8f08
542024-07-25T05:54:53.061ZINFOcrucible-agent (dropshot): request completed latency_us = 713 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38701 req_id = 5bfb1ad1-30f1-4e1c-b9ee-29c1927cccfa response_code = 200 uri = /crucible/0/regions
552024-07-25T05:54:53.175ZINFOcrucible-agent (worker): region files created ok region = a035f0b9-ad4e-4c2d-a88e-dc8d80cc8f08
562024-07-25T05:54:53.175ZINFOcrucible-agent (datafile): region a035f0b9-ad4e-4c2d-a88e-dc8d80cc8f08 state: Requested -> Created
572024-07-25T05:54:53.175ZINFOcrucible-agent (worker): applying SMF actions post create...
582024-07-25T05:54:53.176ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-a035f0b9-ad4e-4c2d-a88e-dc8d80cc8f08
592024-07-25T05:54:53.195ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-a035f0b9-ad4e-4c2d-a88e-dc8d80cc8f08
602024-07-25T05:54:53.263ZINFOcrucible-agent (worker): creating config property group
612024-07-25T05:54:53.292ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-a035f0b9-ad4e-4c2d-a88e-dc8d80cc8f08
622024-07-25T05:54:53.293ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a035f0b9-ad4e-4c2d-a88e-dc8d80cc8f08
632024-07-25T05:54:53.293ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
642024-07-25T05:54:53.293ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
652024-07-25T05:54:53.293ZINFOcrucible-agent (worker): commit
662024-07-25T05:54:53.318ZINFOcrucible-agent (worker): ok!
672024-07-25T05:54:53.332ZINFOcrucible-agent (worker): SMF ok!
682024-07-25T05:54:53.377ZINFOcrucible-agent (dropshot): request completed latency_us = 50 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38701 req_id = 3537d2bc-e7d9-4c22-b833-79fd6a387477 response_code = 200 uri = /crucible/0/regions
692024-07-25T05:54:58.617ZINFOcrucible-agent (datafile): region a952d4a6-69f5-4b67-af88-557bafc5cd46 state: Requested
702024-07-25T05:54:58.617ZINFOcrucible-agent (dropshot): request completed latency_us = 282 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38701 req_id = 1d75a1ec-3238-4446-ba23-f062476c0eaa response_code = 200 uri = /crucible/0/regions
712024-07-25T05:54:58.617ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
722024-07-25T05:54:58.625ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_e43689a1-378e-4be8-9661-3bc522647c6d/crucible/regions/a952d4a6-69f5-4b67-af88-557bafc5cd46
732024-07-25T05:54:58.625ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_e43689a1-378e-4be8-9661-3bc522647c6d/crucible/regions/a952d4a6-69f5-4b67-af88-557bafc5cd46
742024-07-25T05:54:58.709ZINFOcrucible-agent (worker): creating region Region { id: RegionId("a952d4a6-69f5-4b67-af88-557bafc5cd46"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/a952d4a6-69f5-4b67-af88-557bafc5cd46" region = a952d4a6-69f5-4b67-af88-557bafc5cd46
752024-07-25T05:54:58.711ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "a952d4a6-69f5-4b67-af88-557bafc5cd46" "--data" "/data/regions/a952d4a6-69f5-4b67-af88-557bafc5cd46" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = a952d4a6-69f5-4b67-af88-557bafc5cd46
762024-07-25T05:54:58.784ZINFOcrucible-agent (dropshot): request completed latency_us = 710 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38701 req_id = c095bc2f-72c3-4039-bdda-ec23672d2d82 response_code = 200 uri = /crucible/0/regions
772024-07-25T05:54:58.877ZINFOcrucible-agent (worker): region files created ok region = a952d4a6-69f5-4b67-af88-557bafc5cd46
782024-07-25T05:54:58.877ZINFOcrucible-agent (datafile): region a952d4a6-69f5-4b67-af88-557bafc5cd46 state: Requested -> Created
792024-07-25T05:54:58.877ZINFOcrucible-agent (worker): applying SMF actions post create...
802024-07-25T05:54:58.879ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-a952d4a6-69f5-4b67-af88-557bafc5cd46
812024-07-25T05:54:58.904ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-a952d4a6-69f5-4b67-af88-557bafc5cd46
822024-07-25T05:54:58.928ZINFOcrucible-agent (worker): creating config property group
832024-07-25T05:54:58.941ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-a952d4a6-69f5-4b67-af88-557bafc5cd46
842024-07-25T05:54:58.941ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a952d4a6-69f5-4b67-af88-557bafc5cd46
852024-07-25T05:54:58.941ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
862024-07-25T05:54:58.941ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
872024-07-25T05:54:58.941ZINFOcrucible-agent (worker): commit
882024-07-25T05:54:58.958ZINFOcrucible-agent (worker): ok!
892024-07-25T05:54:58.970ZINFOcrucible-agent (worker): SMF ok!
902024-07-25T05:54:59.273ZINFOcrucible-agent (dropshot): request completed latency_us = 79 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38701 req_id = 9c056e23-e2af-4a2b-bcd1-59756fa7f773 response_code = 200 uri = /crucible/0/regions
912024-07-25T05:55:26.583ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:38701 req_id = d50c3bb7-07b4-44a5-a638-06a37f8bd074 response_code = 200 uri = /crucible/0/regions/a952d4a6-69f5-4b67-af88-557bafc5cd46
922024-07-25T05:55:26.585ZINFOcrucible-agent (datafile): path is "/data/regions/a952d4a6-69f5-4b67-af88-557bafc5cd46"
932024-07-25T05:55:26.603ZINFOcrucible-agent (datafile): dataset is oxp_e43689a1-378e-4be8-9661-3bc522647c6d/crucible/regions/a952d4a6-69f5-4b67-af88-557bafc5cd46
942024-07-25T05:55:26.610ZINFOcrucible-agent (datafile): region a952d4a6-69f5-4b67-af88-557bafc5cd46 state: Created -> Tombstoned
952024-07-25T05:55:26.611ZINFOcrucible-agent (worker): applying SMF actions before removal...
962024-07-25T05:55:26.611ZINFOcrucible-agent (dropshot): request completed latency_us = 25615 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:38701 req_id = 6cb8d779-0740-4f7b-97f3-f29e8e1bb5e7 response_code = 204 uri = /crucible/0/regions/a952d4a6-69f5-4b67-af88-557bafc5cd46
972024-07-25T05:55:26.613ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-a952d4a6-69f5-4b67-af88-557bafc5cd46 (instance states: (Some(Online), None))
982024-07-25T05:55:26.614ZINFOcrucible-agent (dropshot): request completed latency_us = 23 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:38701 req_id = 3b8d991e-eebc-480e-bea1-076e3c3534ce response_code = 200 uri = /crucible/0/regions/a952d4a6-69f5-4b67-af88-557bafc5cd46
992024-07-25T05:55:26.626ZINFOcrucible-agent (worker): SMF ok!
1002024-07-25T05:55:26.636ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_e43689a1-378e-4be8-9661-3bc522647c6d/crucible/regions/a952d4a6-69f5-4b67-af88-557bafc5cd46" region = a952d4a6-69f5-4b67-af88-557bafc5cd46
1012024-07-25T05:55:26.724ZINFOcrucible-agent (datafile): region a952d4a6-69f5-4b67-af88-557bafc5cd46 state: Tombstoned -> Destroyed
1022024-07-25T05:55:26.749ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:38701 req_id = a6276d22-fbd2-4d46-a966-d30086148f7d response_code = 200 uri = /crucible/0/regions/a952d4a6-69f5-4b67-af88-557bafc5cd46