Max level shown:
1 [ May 28 17:30:48 Enabled. ]
2 [ May 28 17:30:48 Rereading configuration. ]
3 [ May 28 17:30:51 Rereading configuration. ]
4 [ May 28 17:30:56 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_3bb66df1-b52c-4843-b22b-ff6686fa06b8/crucible -l [fd00:1122:3344:101::1a]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-05-28T17:30:56.421ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: bd9a0e2abe6b6b89aec8c85f4ee57474144ed150 Commit timestamp: 2026-05-14T15:48:00.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-05-28T17:30:56.421ZINFOcrucible-agent: dataset: "oxp_3bb66df1-b52c-4843-b22b-ff6686fa06b8/crucible"
82026-05-28T17:30:56.421ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::1a]:32345
92026-05-28T17:30:56.421ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-05-28T17:30:56.434ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-05-28T17:30:56.501ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::1a]:32345
122026-05-28T17:41:10.461ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::c]:41808
132026-05-28T17:41:10.509ZINFOcrucible-agent (datafile): region 5a621dbc-7302-413c-b565-466a7761b5f6 state: Requested
142026-05-28T17:41:10.515ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152026-05-28T17:41:10.515ZINFOcrucible-agent (dropshot): request completed latency_us = 19071 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:41808 req_id = ec06f2d9-5450-42a9-8986-84c71f14d622 response_code = 200 uri = /crucible/0/regions
162026-05-28T17:41:10.528ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_3bb66df1-b52c-4843-b22b-ff6686fa06b8/crucible/regions/5a621dbc-7302-413c-b565-466a7761b5f6
172026-05-28T17:41:10.528ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_3bb66df1-b52c-4843-b22b-ff6686fa06b8/crucible/regions/5a621dbc-7302-413c-b565-466a7761b5f6
182026-05-28T17:41:10.586ZINFOcrucible-agent (worker): creating region Region { id: RegionId("5a621dbc-7302-413c-b565-466a7761b5f6"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/5a621dbc-7302-413c-b565-466a7761b5f6" region = 5a621dbc-7302-413c-b565-466a7761b5f6
192026-05-28T17:41:10.589ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "5a621dbc-7302-413c-b565-466a7761b5f6" "--data" "/data/regions/5a621dbc-7302-413c-b565-466a7761b5f6" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 5a621dbc-7302-413c-b565-466a7761b5f6
202026-05-28T17:41:10.660ZINFOcrucible-agent (dropshot): request completed latency_us = 44 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:41808 req_id = f55103a9-dc21-4201-b4bc-4e383366b445 response_code = 200 uri = /crucible/0/regions
212026-05-28T17:41:10.950ZINFOcrucible-agent (worker): region files created ok region = 5a621dbc-7302-413c-b565-466a7761b5f6
222026-05-28T17:41:10.950ZINFOcrucible-agent (datafile): region 5a621dbc-7302-413c-b565-466a7761b5f6 state: Requested -> Created
232026-05-28T17:41:10.950ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-05-28T17:41:10.952ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-5a621dbc-7302-413c-b565-466a7761b5f6
252026-05-28T17:41:10.963ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-5a621dbc-7302-413c-b565-466a7761b5f6
262026-05-28T17:41:10.971ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:41808 req_id = 7d55d080-6b17-454c-8563-00902b297e41 response_code = 200 uri = /crucible/0/regions
272026-05-28T17:41:10.981ZINFOcrucible-agent (worker): creating config property group
282026-05-28T17:41:10.984ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5a621dbc-7302-413c-b565-466a7761b5f6
292026-05-28T17:41:10.987ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5a621dbc-7302-413c-b565-466a7761b5f6
302026-05-28T17:41:10.987ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
312026-05-28T17:41:10.988ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
322026-05-28T17:41:10.988ZINFOcrucible-agent (worker): commit
332026-05-28T17:41:11.002ZINFOcrucible-agent (worker): ok!
342026-05-28T17:41:11.006ZINFOcrucible-agent (worker): SMF ok!
352026-05-28T17:41:17.087ZINFOcrucible-agent (datafile): region c573aacd-8e95-4994-81e2-12992df2a446 state: Requested
362026-05-28T17:41:17.087ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
372026-05-28T17:41:17.087ZINFOcrucible-agent (dropshot): request completed latency_us = 268 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:41808 req_id = f05202ab-6bd3-4e38-b75f-ee920423d235 response_code = 200 uri = /crucible/0/regions
382026-05-28T17:41:17.094ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_3bb66df1-b52c-4843-b22b-ff6686fa06b8/crucible/regions/c573aacd-8e95-4994-81e2-12992df2a446
392026-05-28T17:41:17.094ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_3bb66df1-b52c-4843-b22b-ff6686fa06b8/crucible/regions/c573aacd-8e95-4994-81e2-12992df2a446
402026-05-28T17:41:17.144ZINFOcrucible-agent (worker): creating region Region { id: RegionId("c573aacd-8e95-4994-81e2-12992df2a446"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/c573aacd-8e95-4994-81e2-12992df2a446" region = c573aacd-8e95-4994-81e2-12992df2a446
412026-05-28T17:41:17.144ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "c573aacd-8e95-4994-81e2-12992df2a446" "--data" "/data/regions/c573aacd-8e95-4994-81e2-12992df2a446" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = c573aacd-8e95-4994-81e2-12992df2a446
422026-05-28T17:41:17.213ZINFOcrucible-agent (worker): region files created ok region = c573aacd-8e95-4994-81e2-12992df2a446
432026-05-28T17:41:17.214ZINFOcrucible-agent (datafile): region c573aacd-8e95-4994-81e2-12992df2a446 state: Requested -> Created
442026-05-28T17:41:17.214ZINFOcrucible-agent (worker): applying SMF actions post create...
452026-05-28T17:41:17.215ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/5a621dbc-7302-413c-b565-466a7761b5f6
462026-05-28T17:41:17.215ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
472026-05-28T17:41:17.215ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::1a
482026-05-28T17:41:17.215ZINFOcrucible-agent (worker): reconfig required
492026-05-28T17:41:17.215ZINFOcrucible-agent (worker): using existing config property group
502026-05-28T17:41:17.215ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5a621dbc-7302-413c-b565-466a7761b5f6
512026-05-28T17:41:17.215ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5a621dbc-7302-413c-b565-466a7761b5f6
522026-05-28T17:41:17.215ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
532026-05-28T17:41:17.215ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
542026-05-28T17:41:17.215ZINFOcrucible-agent (worker): commit
552026-05-28T17:41:17.219ZINFOcrucible-agent (worker): ok!
562026-05-28T17:41:17.219ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-c573aacd-8e95-4994-81e2-12992df2a446
572026-05-28T17:41:17.222ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-c573aacd-8e95-4994-81e2-12992df2a446
582026-05-28T17:41:17.229ZINFOcrucible-agent (worker): creating config property group
592026-05-28T17:41:17.231ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-c573aacd-8e95-4994-81e2-12992df2a446
602026-05-28T17:41:17.232ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c573aacd-8e95-4994-81e2-12992df2a446
612026-05-28T17:41:17.232ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
622026-05-28T17:41:17.232ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
632026-05-28T17:41:17.232ZINFOcrucible-agent (worker): commit
642026-05-28T17:41:17.235ZINFOcrucible-agent (worker): ok!
652026-05-28T17:41:17.240ZINFOcrucible-agent (worker): SMF ok!
662026-05-28T17:41:17.386ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:41808 req_id = 47a2b2c4-855d-4b4c-9cf1-19747ddfef9a response_code = 200 uri = /crucible/0/regions
672026-05-28T17:41:41.506ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:41808 req_id = ae1e9844-3648-495e-96a4-4478d3555c47 response_code = 200 uri = /crucible/0/regions/c573aacd-8e95-4994-81e2-12992df2a446
682026-05-28T17:41:41.508ZINFOcrucible-agent (datafile): path is "/data/regions/c573aacd-8e95-4994-81e2-12992df2a446"
692026-05-28T17:41:41.518ZINFOcrucible-agent (datafile): dataset is oxp_3bb66df1-b52c-4843-b22b-ff6686fa06b8/crucible/regions/c573aacd-8e95-4994-81e2-12992df2a446
702026-05-28T17:41:41.527ZINFOcrucible-agent (datafile): region c573aacd-8e95-4994-81e2-12992df2a446 state: Created -> Tombstoned
712026-05-28T17:41:41.527ZINFOcrucible-agent (worker): applying SMF actions before removal...
722026-05-28T17:41:41.527ZINFOcrucible-agent (dropshot): request completed latency_us = 19080 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:41808 req_id = acac099d-9321-484f-9675-85ef65a3e449 response_code = 204 uri = /crucible/0/regions/c573aacd-8e95-4994-81e2-12992df2a446
732026-05-28T17:41:41.528ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-c573aacd-8e95-4994-81e2-12992df2a446 (instance states: (Some(Online), None))
742026-05-28T17:41:41.531ZINFOcrucible-agent (dropshot): request completed latency_us = 32 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:41808 req_id = c5287003-535d-4514-bdf8-5c0e41324bb7 response_code = 200 uri = /crucible/0/regions/c573aacd-8e95-4994-81e2-12992df2a446
752026-05-28T17:41:41.536ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/5a621dbc-7302-413c-b565-466a7761b5f6
762026-05-28T17:41:41.536ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
772026-05-28T17:41:41.536ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::1a
782026-05-28T17:41:41.536ZINFOcrucible-agent (worker): reconfig required
792026-05-28T17:41:41.536ZINFOcrucible-agent (worker): using existing config property group
802026-05-28T17:41:41.536ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5a621dbc-7302-413c-b565-466a7761b5f6
812026-05-28T17:41:41.536ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5a621dbc-7302-413c-b565-466a7761b5f6
822026-05-28T17:41:41.536ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
832026-05-28T17:41:41.536ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
842026-05-28T17:41:41.536ZINFOcrucible-agent (worker): commit
852026-05-28T17:41:41.540ZINFOcrucible-agent (worker): ok!
862026-05-28T17:41:41.540ZINFOcrucible-agent (worker): SMF ok!
872026-05-28T17:41:41.549ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_3bb66df1-b52c-4843-b22b-ff6686fa06b8/crucible/regions/c573aacd-8e95-4994-81e2-12992df2a446" region = c573aacd-8e95-4994-81e2-12992df2a446
882026-05-28T17:41:41.598ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:41808 req_id = dcc92e65-ff03-4e9d-9314-f008fb93609f response_code = 200 uri = /crucible/0/regions/c573aacd-8e95-4994-81e2-12992df2a446
892026-05-28T17:41:41.605ZINFOcrucible-agent (datafile): region c573aacd-8e95-4994-81e2-12992df2a446 state: Tombstoned -> Destroyed
902026-05-28T17:41:41.767ZINFOcrucible-agent (dropshot): request completed latency_us = 39 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:41808 req_id = a07a0c16-516b-4a18-a116-a4999f98c25b response_code = 200 uri = /crucible/0/regions/c573aacd-8e95-4994-81e2-12992df2a446