Max level shown:
1 [ Jul 30 01:30:26 Enabled. ]
2 [ Jul 30 01:30:26 Rereading configuration. ]
3 [ Jul 30 01:30:27 Rereading configuration. ]
4 [ Jul 30 01:30:29 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_0cb06715-9f13-41d3-ac8d-a156c2f253a3/crucible -l [fd00:1122:3344:101::14]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-07-30T01:30:29.699ZINFOcrucible-agent: dataset: "oxp_0cb06715-9f13-41d3-ac8d-a156c2f253a3/crucible"
72024-07-30T01:30:29.700ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::14]:32345
82024-07-30T01:30:29.700ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-07-30T01:30:29.717ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-07-30T01:30:29.784ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::14]:32345
112024-07-30T01:31:46.462ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:61778
122024-07-30T01:31:46.483ZINFOcrucible-agent (datafile): region c358610c-c8b4-4495-a3ee-500720417559 state: Requested
132024-07-30T01:31:46.488ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
142024-07-30T01:31:46.488ZINFOcrucible-agent (dropshot): request completed latency_us = 13588 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61778 req_id = c6f91de7-62ce-4064-a7fc-b5848330bb3f response_code = 200 uri = /crucible/0/regions
152024-07-30T01:31:46.501ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_0cb06715-9f13-41d3-ac8d-a156c2f253a3/crucible/regions/c358610c-c8b4-4495-a3ee-500720417559
162024-07-30T01:31:46.501ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_0cb06715-9f13-41d3-ac8d-a156c2f253a3/crucible/regions/c358610c-c8b4-4495-a3ee-500720417559
172024-07-30T01:31:46.576ZINFOcrucible-agent (worker): creating region Region { id: RegionId("c358610c-c8b4-4495-a3ee-500720417559"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/c358610c-c8b4-4495-a3ee-500720417559" region = c358610c-c8b4-4495-a3ee-500720417559
182024-07-30T01:31:46.578ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "c358610c-c8b4-4495-a3ee-500720417559" "--data" "/data/regions/c358610c-c8b4-4495-a3ee-500720417559" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = c358610c-c8b4-4495-a3ee-500720417559
192024-07-30T01:31:46.743ZINFOcrucible-agent (worker): region files created ok region = c358610c-c8b4-4495-a3ee-500720417559
202024-07-30T01:31:46.743ZINFOcrucible-agent (datafile): region c358610c-c8b4-4495-a3ee-500720417559 state: Requested -> Created
212024-07-30T01:31:46.743ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-07-30T01:31:46.743ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-c358610c-c8b4-4495-a3ee-500720417559
232024-07-30T01:31:46.748ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-c358610c-c8b4-4495-a3ee-500720417559
242024-07-30T01:31:46.762ZINFOcrucible-agent (worker): creating config property group
252024-07-30T01:31:46.765ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-c358610c-c8b4-4495-a3ee-500720417559
262024-07-30T01:31:46.773ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c358610c-c8b4-4495-a3ee-500720417559
272024-07-30T01:31:46.775ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-07-30T01:31:46.775ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
292024-07-30T01:31:46.776ZINFOcrucible-agent (worker): commit
302024-07-30T01:31:46.779ZINFOcrucible-agent (worker): ok!
312024-07-30T01:31:46.783ZINFOcrucible-agent (worker): SMF ok!
322024-07-30T01:31:46.864ZINFOcrucible-agent (dropshot): request completed latency_us = 733 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61778 req_id = e42c64d1-b5cd-4373-a5f4-e2d7ee34c2f4 response_code = 200 uri = /crucible/0/regions
332024-07-30T01:37:06.812ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:33597
342024-07-30T01:37:06.812ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:33597 req_id = 7978a0a8-bc62-49c2-9109-7d8c44a2a441 response_code = 200 uri = /crucible/0/regions/c358610c-c8b4-4495-a3ee-500720417559
352024-07-30T01:37:06.815ZINFOcrucible-agent (datafile): path is "/data/regions/c358610c-c8b4-4495-a3ee-500720417559"
362024-07-30T01:37:06.824ZINFOcrucible-agent (datafile): dataset is oxp_0cb06715-9f13-41d3-ac8d-a156c2f253a3/crucible/regions/c358610c-c8b4-4495-a3ee-500720417559
372024-07-30T01:37:06.833ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_0cb06715-9f13-41d3-ac8d-a156c2f253a3/crucible/regions/c358610c-c8b4-4495-a3ee-500720417559@f5199ea3-dcee-4c01-98b8-ff5c9bf72a4b
382024-07-30T01:37:06.833ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_0cb06715-9f13-41d3-ac8d-a156c2f253a3/crucible/regions/c358610c-c8b4-4495-a3ee-500720417559", "f5199ea3-dcee-4c01-98b8-ff5c9bf72a4b"]
392024-07-30T01:37:06.844ZINFOcrucible-agent (dropshot): request completed latency_us = 32231 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:33597 req_id = 825a5b50-d05d-409b-828d-951386b18ff3 response_code = 200 uri = /crucible/0/regions/c358610c-c8b4-4495-a3ee-500720417559/snapshots/f5199ea3-dcee-4c01-98b8-ff5c9bf72a4b
402024-07-30T01:37:06.845ZINFOcrucible-agent (datafile): path is "/data/regions/c358610c-c8b4-4495-a3ee-500720417559"
412024-07-30T01:37:06.853ZINFOcrucible-agent (datafile): dataset is oxp_0cb06715-9f13-41d3-ac8d-a156c2f253a3/crucible/regions/c358610c-c8b4-4495-a3ee-500720417559
422024-07-30T01:37:06.862ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_0cb06715-9f13-41d3-ac8d-a156c2f253a3/crucible/regions/c358610c-c8b4-4495-a3ee-500720417559@f5199ea3-dcee-4c01-98b8-ff5c9bf72a4b
432024-07-30T01:37:06.862ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_0cb06715-9f13-41d3-ac8d-a156c2f253a3/crucible/regions/c358610c-c8b4-4495-a3ee-500720417559", "f5199ea3-dcee-4c01-98b8-ff5c9bf72a4b"]
442024-07-30T01:37:06.870ZINFOcrucible-agent (datafile): requesting running snapshot c358610c-c8b4-4495-a3ee-500720417559-f5199ea3-dcee-4c01-98b8-ff5c9bf72a4b state: Requested
452024-07-30T01:37:06.871ZINFOcrucible-agent (worker): applying SMF actions for region c358610c-c8b4-4495-a3ee-500720417559 running snapshot f5199ea3-dcee-4c01-98b8-ff5c9bf72a4b (state Requested)...
462024-07-30T01:37:06.871ZINFOcrucible-agent (dropshot): request completed latency_us = 26341 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33597 req_id = 85b1d79c-9826-4249-ad47-17fed4907ec2 response_code = 200 uri = /crucible/0/regions/c358610c-c8b4-4495-a3ee-500720417559/snapshots/f5199ea3-dcee-4c01-98b8-ff5c9bf72a4b/run
472024-07-30T01:37:06.877ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/c358610c-c8b4-4495-a3ee-500720417559
482024-07-30T01:37:06.877ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-07-30T01:37:06.877ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
502024-07-30T01:37:06.877ZINFOcrucible-agent (worker): reconfig required
512024-07-30T01:37:06.877ZINFOcrucible-agent (worker): using existing config property group
522024-07-30T01:37:06.877ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-c358610c-c8b4-4495-a3ee-500720417559
532024-07-30T01:37:06.877ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c358610c-c8b4-4495-a3ee-500720417559
542024-07-30T01:37:06.877ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-07-30T01:37:06.877ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
562024-07-30T01:37:06.877ZINFOcrucible-agent (worker): commit
572024-07-30T01:37:06.885ZINFOcrucible-agent (worker): ok!
582024-07-30T01:37:06.885ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-c358610c-c8b4-4495-a3ee-500720417559-f5199ea3-dcee-4c01-98b8-ff5c9bf72a4b
592024-07-30T01:37:06.891ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-c358610c-c8b4-4495-a3ee-500720417559-f5199ea3-dcee-4c01-98b8-ff5c9bf72a4b
602024-07-30T01:37:06.902ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-c358610c-c8b4-4495-a3ee-500720417559-f5199ea3-dcee-4c01-98b8-ff5c9bf72a4b
612024-07-30T01:37:06.903ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c358610c-c8b4-4495-a3ee-500720417559/.zfs/snapshot/f5199ea3-dcee-4c01-98b8-ff5c9bf72a4b
622024-07-30T01:37:06.903ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
632024-07-30T01:37:06.903ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-07-30T01:37:06.903ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
652024-07-30T01:37:06.903ZINFOcrucible-agent (worker): commit
662024-07-30T01:37:06.907ZINFOcrucible-agent (worker): ok!
672024-07-30T01:37:06.911ZINFOcrucible-agent (worker): SMF ok!
682024-07-30T01:37:06.911ZINFOcrucible-agent (datafile): region c358610c-c8b4-4495-a3ee-500720417559 running snapshot f5199ea3-dcee-4c01-98b8-ff5c9bf72a4b state: Requested -> Created
692024-07-30T01:37:09.800ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:62062
702024-07-30T01:37:09.801ZINFOcrucible-agent (datafile): region 13190d17-ff58-4cdc-878d-9a10c2eb1a85 state: Requested
712024-07-30T01:37:09.801ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
722024-07-30T01:37:09.801ZINFOcrucible-agent (dropshot): request completed latency_us = 485 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:62062 req_id = 1305ef86-ea17-411d-82c9-c0eb56d4f7d4 response_code = 200 uri = /crucible/0/regions
732024-07-30T01:37:09.812ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_0cb06715-9f13-41d3-ac8d-a156c2f253a3/crucible/regions/13190d17-ff58-4cdc-878d-9a10c2eb1a85
742024-07-30T01:37:09.812ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_0cb06715-9f13-41d3-ac8d-a156c2f253a3/crucible/regions/13190d17-ff58-4cdc-878d-9a10c2eb1a85
752024-07-30T01:37:09.880ZINFOcrucible-agent (worker): creating region Region { id: RegionId("13190d17-ff58-4cdc-878d-9a10c2eb1a85"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/13190d17-ff58-4cdc-878d-9a10c2eb1a85" region = 13190d17-ff58-4cdc-878d-9a10c2eb1a85
762024-07-30T01:37:09.880ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "13190d17-ff58-4cdc-878d-9a10c2eb1a85" "--data" "/data/regions/13190d17-ff58-4cdc-878d-9a10c2eb1a85" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 13190d17-ff58-4cdc-878d-9a10c2eb1a85
772024-07-30T01:37:09.943ZINFOcrucible-agent (worker): region files created ok region = 13190d17-ff58-4cdc-878d-9a10c2eb1a85
782024-07-30T01:37:09.943ZINFOcrucible-agent (datafile): region 13190d17-ff58-4cdc-878d-9a10c2eb1a85 state: Requested -> Created
792024-07-30T01:37:09.943ZINFOcrucible-agent (worker): applying SMF actions post create...
802024-07-30T01:37:09.943ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-13190d17-ff58-4cdc-878d-9a10c2eb1a85
812024-07-30T01:37:09.946ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-13190d17-ff58-4cdc-878d-9a10c2eb1a85
822024-07-30T01:37:09.953ZINFOcrucible-agent (worker): creating config property group
832024-07-30T01:37:09.956ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-13190d17-ff58-4cdc-878d-9a10c2eb1a85
842024-07-30T01:37:09.956ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/13190d17-ff58-4cdc-878d-9a10c2eb1a85
852024-07-30T01:37:09.956ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
862024-07-30T01:37:09.956ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
872024-07-30T01:37:09.956ZINFOcrucible-agent (worker): commit
882024-07-30T01:37:09.961ZINFOcrucible-agent (worker): ok!
892024-07-30T01:37:09.966ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/c358610c-c8b4-4495-a3ee-500720417559
902024-07-30T01:37:09.966ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
912024-07-30T01:37:09.966ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
922024-07-30T01:37:09.966ZINFOcrucible-agent (worker): reconfig required
932024-07-30T01:37:09.966ZINFOcrucible-agent (worker): using existing config property group
942024-07-30T01:37:09.966ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-c358610c-c8b4-4495-a3ee-500720417559
952024-07-30T01:37:09.970ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c358610c-c8b4-4495-a3ee-500720417559
962024-07-30T01:37:09.970ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
972024-07-30T01:37:09.970ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
982024-07-30T01:37:09.970ZINFOcrucible-agent (worker): commit
992024-07-30T01:37:09.973ZINFOcrucible-agent (worker): ok!
1002024-07-30T01:37:09.975ZINFOcrucible-agent (worker): SMF ok!
1012024-07-30T01:37:10.148ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:62062 req_id = 47419b42-2037-4d24-a54f-70526b8decec response_code = 200 uri = /crucible/0/regions
1022024-07-30T01:37:33.880ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:62062 req_id = 488413b0-7a14-41e1-a3b3-35a84d3f7a82 response_code = 200 uri = /crucible/0/regions/13190d17-ff58-4cdc-878d-9a10c2eb1a85
1032024-07-30T01:37:33.884ZINFOcrucible-agent (datafile): path is "/data/regions/13190d17-ff58-4cdc-878d-9a10c2eb1a85"
1042024-07-30T01:37:33.894ZINFOcrucible-agent (datafile): dataset is oxp_0cb06715-9f13-41d3-ac8d-a156c2f253a3/crucible/regions/13190d17-ff58-4cdc-878d-9a10c2eb1a85
1052024-07-30T01:37:33.903ZINFOcrucible-agent (datafile): region 13190d17-ff58-4cdc-878d-9a10c2eb1a85 state: Created -> Tombstoned
1062024-07-30T01:37:33.903ZINFOcrucible-agent (worker): applying SMF actions before removal...
1072024-07-30T01:37:33.903ZINFOcrucible-agent (dropshot): request completed latency_us = 19736 local_addr = [fd00:1122:3344:101::14]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:62062 req_id = 633701d9-3e7f-4a1d-93f4-73f0b17ff085 response_code = 204 uri = /crucible/0/regions/13190d17-ff58-4cdc-878d-9a10c2eb1a85
1082024-07-30T01:37:33.904ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-13190d17-ff58-4cdc-878d-9a10c2eb1a85 (instance states: (Some(Online), None))
1092024-07-30T01:37:33.907ZINFOcrucible-agent (dropshot): request completed latency_us = 729 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:62062 req_id = 4cb495f7-fe51-4f42-b885-81ee71783016 response_code = 200 uri = /crucible/0/regions/13190d17-ff58-4cdc-878d-9a10c2eb1a85
1102024-07-30T01:37:33.909ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/c358610c-c8b4-4495-a3ee-500720417559
1112024-07-30T01:37:33.909ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1122024-07-30T01:37:33.909ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
1132024-07-30T01:37:33.909ZINFOcrucible-agent (worker): reconfig required
1142024-07-30T01:37:33.909ZINFOcrucible-agent (worker): using existing config property group
1152024-07-30T01:37:33.909ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-c358610c-c8b4-4495-a3ee-500720417559
1162024-07-30T01:37:33.909ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c358610c-c8b4-4495-a3ee-500720417559
1172024-07-30T01:37:33.909ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1182024-07-30T01:37:33.910ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
1192024-07-30T01:37:33.910ZINFOcrucible-agent (worker): commit
1202024-07-30T01:37:33.913ZINFOcrucible-agent (worker): ok!
1212024-07-30T01:37:33.914ZINFOcrucible-agent (worker): SMF ok!
1222024-07-30T01:37:33.924ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_0cb06715-9f13-41d3-ac8d-a156c2f253a3/crucible/regions/13190d17-ff58-4cdc-878d-9a10c2eb1a85" region = 13190d17-ff58-4cdc-878d-9a10c2eb1a85
1232024-07-30T01:37:33.972ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:62062 req_id = ec1a96d6-d235-4eff-af96-723a497c2afc response_code = 200 uri = /crucible/0/regions/13190d17-ff58-4cdc-878d-9a10c2eb1a85
1242024-07-30T01:37:33.975ZINFOcrucible-agent (datafile): region 13190d17-ff58-4cdc-878d-9a10c2eb1a85 state: Tombstoned -> Destroyed
1252024-07-30T01:37:34.050ZINFOcrucible-agent (dropshot): request completed latency_us = 20 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:62062 req_id = 48cde344-024d-4865-9c28-77c328e3ce66 response_code = 200 uri = /crucible/0/regions/13190d17-ff58-4cdc-878d-9a10c2eb1a85