Max level shown:
1 [ Jan 9 01:03:32 Enabled. ]
2 [ Jan 9 01:03:32 Rereading configuration. ]
3 [ Jan 9 01:03:34 Rereading configuration. ]
4 [ Jan 9 01:03:38 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_4eca6aeb-4f07-4f83-9a97-c0923ef06389/crucible -l [fd00:1122:3344:101::12]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-01-09T01:03:38.752ZINFOcrucible-agent: dataset: "oxp_4eca6aeb-4f07-4f83-9a97-c0923ef06389/crucible"
72025-01-09T01:03:38.755ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::12]:32345
82025-01-09T01:03:38.755ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92025-01-09T01:03:38.773ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102025-01-09T01:03:38.891ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::12]:32345
112025-01-09T01:04:01.565ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:46255
122025-01-09T01:04:01.604ZINFOcrucible-agent (datafile): region 0ea978d6-db3f-44e4-803a-4a331c78f132 state: Requested
132025-01-09T01:04:01.608ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
142025-01-09T01:04:01.613ZINFOcrucible-agent (dropshot): request completed latency_us = 19677 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46255 req_id = 3b6ac334-f7d7-46fa-a27f-b4e3689053eb response_code = 200 uri = /crucible/0/regions
152025-01-09T01:04:01.619ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_4eca6aeb-4f07-4f83-9a97-c0923ef06389/crucible/regions/0ea978d6-db3f-44e4-803a-4a331c78f132
162025-01-09T01:04:01.619ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_4eca6aeb-4f07-4f83-9a97-c0923ef06389/crucible/regions/0ea978d6-db3f-44e4-803a-4a331c78f132
172025-01-09T01:04:01.768ZINFOcrucible-agent (worker): creating region Region { id: RegionId("0ea978d6-db3f-44e4-803a-4a331c78f132"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/0ea978d6-db3f-44e4-803a-4a331c78f132" region = 0ea978d6-db3f-44e4-803a-4a331c78f132
182025-01-09T01:04:01.770ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "0ea978d6-db3f-44e4-803a-4a331c78f132" "--data" "/data/regions/0ea978d6-db3f-44e4-803a-4a331c78f132" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 0ea978d6-db3f-44e4-803a-4a331c78f132
192025-01-09T01:04:01.947ZINFOcrucible-agent (dropshot): request completed latency_us = 72 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46255 req_id = 3b729a1f-fe65-4458-abb7-ed22d03988d0 response_code = 200 uri = /crucible/0/regions
202025-01-09T01:04:02.081ZINFOcrucible-agent (worker): region files created ok region = 0ea978d6-db3f-44e4-803a-4a331c78f132
212025-01-09T01:04:02.081ZINFOcrucible-agent (datafile): region 0ea978d6-db3f-44e4-803a-4a331c78f132 state: Requested -> Created
222025-01-09T01:04:02.081ZINFOcrucible-agent (worker): applying SMF actions post create...
232025-01-09T01:04:02.081ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-0ea978d6-db3f-44e4-803a-4a331c78f132
242025-01-09T01:04:02.097ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-0ea978d6-db3f-44e4-803a-4a331c78f132
252025-01-09T01:04:02.133ZINFOcrucible-agent (worker): creating config property group
262025-01-09T01:04:02.153ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0ea978d6-db3f-44e4-803a-4a331c78f132
272025-01-09T01:04:02.154ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0ea978d6-db3f-44e4-803a-4a331c78f132
282025-01-09T01:04:02.154ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292025-01-09T01:04:02.154ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
302025-01-09T01:04:02.154ZINFOcrucible-agent (worker): commit
312025-01-09T01:04:02.177ZINFOcrucible-agent (worker): ok!
322025-01-09T01:04:02.201ZINFOcrucible-agent (worker): SMF ok!
332025-01-09T01:04:02.206ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:46255 req_id = ff597dc2-741e-4bab-9fba-f3d106d86d54 response_code = 200 uri = /crucible/0/regions
342025-01-09T01:14:09.056ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:57610
352025-01-09T01:14:09.060ZINFOcrucible-agent (dropshot): request completed latency_us = 4128 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57610 req_id = b6eaa11c-c6a2-4438-afe8-1ffe0a73a0cc response_code = 200 uri = /crucible/0/regions/0ea978d6-db3f-44e4-803a-4a331c78f132
362025-01-09T01:14:09.065ZINFOcrucible-agent (datafile): path is "/data/regions/0ea978d6-db3f-44e4-803a-4a331c78f132"
372025-01-09T01:14:09.078ZINFOcrucible-agent (datafile): dataset is oxp_4eca6aeb-4f07-4f83-9a97-c0923ef06389/crucible/regions/0ea978d6-db3f-44e4-803a-4a331c78f132
382025-01-09T01:14:09.087ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_4eca6aeb-4f07-4f83-9a97-c0923ef06389/crucible/regions/0ea978d6-db3f-44e4-803a-4a331c78f132@976d8d11-2930-4c15-8236-b132904f582d
392025-01-09T01:14:09.087ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_4eca6aeb-4f07-4f83-9a97-c0923ef06389/crucible/regions/0ea978d6-db3f-44e4-803a-4a331c78f132", "976d8d11-2930-4c15-8236-b132904f582d"]
402025-01-09T01:14:09.098ZINFOcrucible-agent (dropshot): request completed latency_us = 32593 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57610 req_id = 47fee733-76ed-4964-b2b3-0ce34c7b456e response_code = 200 uri = /crucible/0/regions/0ea978d6-db3f-44e4-803a-4a331c78f132/snapshots/976d8d11-2930-4c15-8236-b132904f582d
412025-01-09T01:14:09.101ZINFOcrucible-agent (datafile): path is "/data/regions/0ea978d6-db3f-44e4-803a-4a331c78f132"
422025-01-09T01:14:09.110ZINFOcrucible-agent (datafile): dataset is oxp_4eca6aeb-4f07-4f83-9a97-c0923ef06389/crucible/regions/0ea978d6-db3f-44e4-803a-4a331c78f132
432025-01-09T01:14:09.119ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_4eca6aeb-4f07-4f83-9a97-c0923ef06389/crucible/regions/0ea978d6-db3f-44e4-803a-4a331c78f132@976d8d11-2930-4c15-8236-b132904f582d
442025-01-09T01:14:09.119ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_4eca6aeb-4f07-4f83-9a97-c0923ef06389/crucible/regions/0ea978d6-db3f-44e4-803a-4a331c78f132", "976d8d11-2930-4c15-8236-b132904f582d"]
452025-01-09T01:14:09.129ZINFOcrucible-agent (datafile): requesting running snapshot 0ea978d6-db3f-44e4-803a-4a331c78f132-976d8d11-2930-4c15-8236-b132904f582d state: Requested
462025-01-09T01:14:09.130ZINFOcrucible-agent (worker): applying SMF actions for region 0ea978d6-db3f-44e4-803a-4a331c78f132 running snapshot 976d8d11-2930-4c15-8236-b132904f582d (state Requested)...
472025-01-09T01:14:09.130ZINFOcrucible-agent (dropshot): request completed latency_us = 29200 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57610 req_id = 433c51b2-2a96-4919-8248-3f89e07522aa response_code = 200 uri = /crucible/0/regions/0ea978d6-db3f-44e4-803a-4a331c78f132/snapshots/976d8d11-2930-4c15-8236-b132904f582d/run
482025-01-09T01:14:09.144ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-0ea978d6-db3f-44e4-803a-4a331c78f132-976d8d11-2930-4c15-8236-b132904f582d
492025-01-09T01:14:09.148ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-0ea978d6-db3f-44e4-803a-4a331c78f132-976d8d11-2930-4c15-8236-b132904f582d
502025-01-09T01:14:09.161ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-0ea978d6-db3f-44e4-803a-4a331c78f132-976d8d11-2930-4c15-8236-b132904f582d
512025-01-09T01:14:09.161ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0ea978d6-db3f-44e4-803a-4a331c78f132/.zfs/snapshot/976d8d11-2930-4c15-8236-b132904f582d
522025-01-09T01:14:09.161ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
532025-01-09T01:14:09.161ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
542025-01-09T01:14:09.161ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
552025-01-09T01:14:09.161ZINFOcrucible-agent (worker): commit
562025-01-09T01:14:09.165ZINFOcrucible-agent (worker): ok!
572025-01-09T01:14:09.169ZINFOcrucible-agent (worker): SMF ok!
582025-01-09T01:14:09.169ZINFOcrucible-agent (datafile): region 0ea978d6-db3f-44e4-803a-4a331c78f132 running snapshot 976d8d11-2930-4c15-8236-b132904f582d state: Requested -> Created
592025-01-09T01:14:09.405ZINFOcrucible-agent (datafile): path is "/data/regions/0ea978d6-db3f-44e4-803a-4a331c78f132"
602025-01-09T01:14:09.415ZINFOcrucible-agent (datafile): dataset is oxp_4eca6aeb-4f07-4f83-9a97-c0923ef06389/crucible/regions/0ea978d6-db3f-44e4-803a-4a331c78f132
612025-01-09T01:14:09.423ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_4eca6aeb-4f07-4f83-9a97-c0923ef06389/crucible/regions/0ea978d6-db3f-44e4-803a-4a331c78f132@976d8d11-2930-4c15-8236-b132904f582d
622025-01-09T01:14:09.423ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_4eca6aeb-4f07-4f83-9a97-c0923ef06389/crucible/regions/0ea978d6-db3f-44e4-803a-4a331c78f132", "976d8d11-2930-4c15-8236-b132904f582d"]
632025-01-09T01:14:09.432ZINFOcrucible-agent (dropshot): request completed latency_us = 26942 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57610 req_id = 4eaf3a37-77bd-4204-b0c3-8d62664ee690 response_code = 200 uri = /crucible/0/regions/0ea978d6-db3f-44e4-803a-4a331c78f132/snapshots/976d8d11-2930-4c15-8236-b132904f582d/run