Max level shown:
1 [ May 12 01:47:05 Enabled. ]
2 [ May 12 01:47:05 Rereading configuration. ]
3 [ May 12 01:47:06 Rereading configuration. ]
4 [ May 12 01:47:10 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_147dfbbd-ffcf-42ba-b2a6-5bff14e21013/crucible -l [fd00:1122:3344:101::17]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62026-05-12T01:47:10.169ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: fefccc0623038371dbded0458f0645bee6b141cb Commit timestamp: 2026-05-11T19:17:35.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72026-05-12T01:47:10.169ZINFOcrucible-agent: dataset: "oxp_147dfbbd-ffcf-42ba-b2a6-5bff14e21013/crucible"
82026-05-12T01:47:10.169ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::17]:32345
92026-05-12T01:47:10.169ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102026-05-12T01:47:10.190ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112026-05-12T01:47:10.262ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::17]:32345
122026-05-12T01:59:29.585ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::c]:45564
132026-05-12T01:59:29.638ZINFOcrucible-agent (datafile): region ff2005a5-0739-4172-b3d6-0f359ea316c3 state: Requested
142026-05-12T01:59:29.645ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152026-05-12T01:59:29.645ZINFOcrucible-agent (dropshot): request completed latency_us = 25829 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:45564 req_id = e122145f-8579-4280-a8bb-f8e309a8495b response_code = 200 uri = /crucible/0/regions
162026-05-12T01:59:29.657ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_147dfbbd-ffcf-42ba-b2a6-5bff14e21013/crucible/regions/ff2005a5-0739-4172-b3d6-0f359ea316c3
172026-05-12T01:59:29.657ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_147dfbbd-ffcf-42ba-b2a6-5bff14e21013/crucible/regions/ff2005a5-0739-4172-b3d6-0f359ea316c3
182026-05-12T01:59:29.708ZINFOcrucible-agent (worker): creating region Region { id: RegionId("ff2005a5-0739-4172-b3d6-0f359ea316c3"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/ff2005a5-0739-4172-b3d6-0f359ea316c3" region = ff2005a5-0739-4172-b3d6-0f359ea316c3
192026-05-12T01:59:29.710ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "ff2005a5-0739-4172-b3d6-0f359ea316c3" "--data" "/data/regions/ff2005a5-0739-4172-b3d6-0f359ea316c3" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = ff2005a5-0739-4172-b3d6-0f359ea316c3
202026-05-12T01:59:29.991ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:45564 req_id = b3944518-b133-4d84-8942-ae1d6eda1dcf response_code = 200 uri = /crucible/0/regions
212026-05-12T01:59:30.040ZINFOcrucible-agent (worker): region files created ok region = ff2005a5-0739-4172-b3d6-0f359ea316c3
222026-05-12T01:59:30.041ZINFOcrucible-agent (datafile): region ff2005a5-0739-4172-b3d6-0f359ea316c3 state: Requested -> Created
232026-05-12T01:59:30.041ZINFOcrucible-agent (worker): applying SMF actions post create...
242026-05-12T01:59:30.043ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-ff2005a5-0739-4172-b3d6-0f359ea316c3
252026-05-12T01:59:30.053ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-ff2005a5-0739-4172-b3d6-0f359ea316c3
262026-05-12T01:59:30.075ZINFOcrucible-agent (worker): creating config property group
272026-05-12T01:59:30.077ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ff2005a5-0739-4172-b3d6-0f359ea316c3
282026-05-12T01:59:30.078ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ff2005a5-0739-4172-b3d6-0f359ea316c3
292026-05-12T01:59:30.078ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302026-05-12T01:59:30.078ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
312026-05-12T01:59:30.078ZINFOcrucible-agent (worker): commit
322026-05-12T01:59:30.094ZINFOcrucible-agent (worker): ok!
332026-05-12T01:59:30.097ZINFOcrucible-agent (worker): SMF ok!
342026-05-12T01:59:30.559ZINFOcrucible-agent (dropshot): request completed latency_us = 72 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:45564 req_id = fe23d498-3fd0-4260-934f-1b3a43562766 response_code = 200 uri = /crucible/0/regions
352026-05-12T01:59:35.940ZINFOcrucible-agent (datafile): region 009f66c8-a9da-46d4-8537-9b6e046aeeb3 state: Requested
362026-05-12T01:59:35.940ZINFOcrucible-agent (dropshot): request completed latency_us = 237 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:45564 req_id = c6347a25-017f-497c-8130-bad03e71bb89 response_code = 200 uri = /crucible/0/regions
372026-05-12T01:59:35.940ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
382026-05-12T01:59:35.947ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_147dfbbd-ffcf-42ba-b2a6-5bff14e21013/crucible/regions/009f66c8-a9da-46d4-8537-9b6e046aeeb3
392026-05-12T01:59:35.947ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_147dfbbd-ffcf-42ba-b2a6-5bff14e21013/crucible/regions/009f66c8-a9da-46d4-8537-9b6e046aeeb3
402026-05-12T01:59:35.984ZINFOcrucible-agent (worker): creating region Region { id: RegionId("009f66c8-a9da-46d4-8537-9b6e046aeeb3"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/009f66c8-a9da-46d4-8537-9b6e046aeeb3" region = 009f66c8-a9da-46d4-8537-9b6e046aeeb3
412026-05-12T01:59:35.984ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "009f66c8-a9da-46d4-8537-9b6e046aeeb3" "--data" "/data/regions/009f66c8-a9da-46d4-8537-9b6e046aeeb3" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 009f66c8-a9da-46d4-8537-9b6e046aeeb3
422026-05-12T01:59:36.034ZINFOcrucible-agent (worker): region files created ok region = 009f66c8-a9da-46d4-8537-9b6e046aeeb3
432026-05-12T01:59:36.034ZINFOcrucible-agent (datafile): region 009f66c8-a9da-46d4-8537-9b6e046aeeb3 state: Requested -> Created
442026-05-12T01:59:36.035ZINFOcrucible-agent (worker): applying SMF actions post create...
452026-05-12T01:59:36.035ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-009f66c8-a9da-46d4-8537-9b6e046aeeb3
462026-05-12T01:59:36.037ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-009f66c8-a9da-46d4-8537-9b6e046aeeb3
472026-05-12T01:59:36.041ZINFOcrucible-agent (worker): creating config property group
482026-05-12T01:59:36.042ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-009f66c8-a9da-46d4-8537-9b6e046aeeb3
492026-05-12T01:59:36.043ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/009f66c8-a9da-46d4-8537-9b6e046aeeb3
502026-05-12T01:59:36.043ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
512026-05-12T01:59:36.043ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
522026-05-12T01:59:36.043ZINFOcrucible-agent (worker): commit
532026-05-12T01:59:36.045ZINFOcrucible-agent (worker): ok!
542026-05-12T01:59:36.048ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ff2005a5-0739-4172-b3d6-0f359ea316c3
552026-05-12T01:59:36.049ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
562026-05-12T01:59:36.049ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::17
572026-05-12T01:59:36.049ZINFOcrucible-agent (worker): reconfig required
582026-05-12T01:59:36.049ZINFOcrucible-agent (worker): using existing config property group
592026-05-12T01:59:36.049ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ff2005a5-0739-4172-b3d6-0f359ea316c3
602026-05-12T01:59:36.049ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ff2005a5-0739-4172-b3d6-0f359ea316c3
612026-05-12T01:59:36.049ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
622026-05-12T01:59:36.049ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
632026-05-12T01:59:36.049ZINFOcrucible-agent (worker): commit
642026-05-12T01:59:36.053ZINFOcrucible-agent (worker): ok!
652026-05-12T01:59:36.054ZINFOcrucible-agent (worker): SMF ok!
662026-05-12T01:59:36.168ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:45564 req_id = 9c157d98-e0cd-4624-8f46-ab3231be6db7 response_code = 200 uri = /crucible/0/regions
672026-05-12T02:00:00.889ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:45564 req_id = f66fdd65-a2f6-4f8c-b90a-1f83646ecdd3 response_code = 200 uri = /crucible/0/regions/009f66c8-a9da-46d4-8537-9b6e046aeeb3
682026-05-12T02:00:00.893ZINFOcrucible-agent (datafile): path is "/data/regions/009f66c8-a9da-46d4-8537-9b6e046aeeb3"
692026-05-12T02:00:00.906ZINFOcrucible-agent (datafile): dataset is oxp_147dfbbd-ffcf-42ba-b2a6-5bff14e21013/crucible/regions/009f66c8-a9da-46d4-8537-9b6e046aeeb3
702026-05-12T02:00:00.914ZINFOcrucible-agent (datafile): region 009f66c8-a9da-46d4-8537-9b6e046aeeb3 state: Created -> Tombstoned
712026-05-12T02:00:00.914ZINFOcrucible-agent (worker): applying SMF actions before removal...
722026-05-12T02:00:00.914ZINFOcrucible-agent (dropshot): request completed latency_us = 21114 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:45564 req_id = 42d06b15-1ad6-4b5f-a52a-61fe2f461bba response_code = 204 uri = /crucible/0/regions/009f66c8-a9da-46d4-8537-9b6e046aeeb3
732026-05-12T02:00:00.916ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-009f66c8-a9da-46d4-8537-9b6e046aeeb3 (instance states: (Some(Online), None))
742026-05-12T02:00:00.918ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/ff2005a5-0739-4172-b3d6-0f359ea316c3
752026-05-12T02:00:00.919ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
762026-05-12T02:00:00.919ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::17
772026-05-12T02:00:00.919ZINFOcrucible-agent (worker): reconfig required
782026-05-12T02:00:00.919ZINFOcrucible-agent (worker): using existing config property group
792026-05-12T02:00:00.919ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ff2005a5-0739-4172-b3d6-0f359ea316c3
802026-05-12T02:00:00.919ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ff2005a5-0739-4172-b3d6-0f359ea316c3
812026-05-12T02:00:00.919ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
822026-05-12T02:00:00.919ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
832026-05-12T02:00:00.919ZINFOcrucible-agent (worker): commit
842026-05-12T02:00:00.921ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:45564 req_id = 9166cd41-1d74-4776-9e28-9355928dd00f response_code = 200 uri = /crucible/0/regions/009f66c8-a9da-46d4-8537-9b6e046aeeb3
852026-05-12T02:00:00.922ZINFOcrucible-agent (worker): ok!
862026-05-12T02:00:00.922ZINFOcrucible-agent (worker): SMF ok!
872026-05-12T02:00:00.934ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_147dfbbd-ffcf-42ba-b2a6-5bff14e21013/crucible/regions/009f66c8-a9da-46d4-8537-9b6e046aeeb3" region = 009f66c8-a9da-46d4-8537-9b6e046aeeb3
882026-05-12T02:00:00.999ZINFOcrucible-agent (datafile): region 009f66c8-a9da-46d4-8537-9b6e046aeeb3 state: Tombstoned -> Destroyed
892026-05-12T02:00:01.042ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:45564 req_id = c211fc17-4046-4abc-9bb2-a1287e95c430 response_code = 200 uri = /crucible/0/regions/009f66c8-a9da-46d4-8537-9b6e046aeeb3