Max level shown:
1 [ Dec 30 06:22:42 Enabled. ]
2 [ Dec 30 06:22:42 Rereading configuration. ]
3 [ Dec 30 06:22:44 Rereading configuration. ]
4 [ Dec 30 06:22:46 Executing start method ("/opt/oxide/lib/svc/manifest/crucible/agent.sh"). ]
5 + . /lib/svc/share/smf_include.sh
6 ++ SMF_EXIT_OK=0
7 ++ SMF_EXIT_NODAEMON=94
8 ++ SMF_EXIT_ERR_FATAL=95
9 ++ SMF_EXIT_ERR_CONFIG=96
10 ++ SMF_EXIT_MON_DEGRADE=97
11 ++ SMF_EXIT_MON_OFFLINE=98
12 ++ SMF_EXIT_ERR_NOSMF=99
13 ++ SMF_EXIT_ERR_PERM=100
14 ++ svcprop -c -p config/datalink svc:/oxide/crucible/agent:default
15 + DATALINK=oxControlService19
16 ++ svcprop -c -p config/gateway svc:/oxide/crucible/agent:default
17 + GATEWAY=fd00:1122:3344:101::1
18 ++ svcprop -c -p config/dataset svc:/oxide/crucible/agent:default
19 + DATASET=oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible
20 ++ svcprop -c -p config/listen_addr svc:/oxide/crucible/agent:default
21 + LISTEN_ADDR=fd00:1122:3344:101::19
22 ++ svcprop -c -p config/listen_port svc:/oxide/crucible/agent:default
23 + LISTEN_PORT=32345
24 ++ svcprop -c -p config/portbase svc:/oxide/crucible/agent:default
25 + PORTBASE=19000
26 ++ svcprop -c -p config/downstairs_prefix svc:/oxide/crucible/agent:default
27 + DOWNSTAIRS_PREFIX=downstairs
28 ++ svcprop -c -p config/snapshot_prefix svc:/oxide/crucible/agent:default
29 + SNAPSHOT_PREFIX=snapshot
30 + [[ oxControlService19 == unknown ]]
31 + [[ fd00:1122:3344:101::1 == unknown ]]
32 + ipadm delete-if oxControlService19
33 ipadm: Could not delete oxControlService19: Interface does not exist
34 + true
35 + ipadm create-if -t oxControlService19
36 + ipadm set-ifprop -t -p mtu=9000 -m ipv4 oxControlService19
37 + ipadm set-ifprop -t -p mtu=9000 -m ipv6 oxControlService19
38 + ipadm show-addr oxControlService19/ll
39 ipadm: Address object not found
40 + ipadm create-addr -t -T addrconf oxControlService19/ll
41 + ipadm show-addr oxControlService19/omicron6
42 ipadm: Address object not found
43 + ipadm create-addr -t -T static -a fd00:1122:3344:101::19 oxControlService19/omicron6
44 + route get -inet6 default -inet6 fd00:1122:3344:101::1
45 default: not in table
46 + route add -inet6 default -inet6 fd00:1122:3344:101::1
47 add net default: gateway fd00:1122:3344:101::1
48 + args=('-D' '/opt/oxide/crucible/bin/crucible-downstairs' '--dataset' "$DATASET" '-l' "[$LISTEN_ADDR]:$LISTEN_PORT" '-P' "$PORTBASE" '-p' "$DOWNSTAIRS_PREFIX" '-s' "$SNAPSHOT_PREFIX")
49 + exec /opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible -l '[fd00:1122:3344:101::19]:32345' -P 19000 -p downstairs -s snapshot
50 note: configured to log to "/dev/stdout"
512023-12-30T06:22:47.542ZINFOcrucible-agent: dataset: "oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible"
522023-12-30T06:22:47.545ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::19]:32345
532023-12-30T06:22:47.545ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
542023-12-30T06:22:47.742ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::19]:32345
552023-12-30T06:23:53.567ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::b]:56260
562023-12-30T06:23:53.639ZINFOcrucible-agent (datafile): region 7b95296d-4d6b-4d14-83f9-428dbd56abfd state: Requested
572023-12-30T06:23:53.640ZINFOcrucible-agent (dropshot): request completed latency_us = 10121 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:56260 req_id = 48b8bf8f-f05d-4c86-931c-a883b6a415f9 response_code = 200 uri = /crucible/0/regions
582023-12-30T06:23:53.640ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
592023-12-30T06:23:53.650ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/7b95296d-4d6b-4d14-83f9-428dbd56abfd
602023-12-30T06:23:53.650ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/7b95296d-4d6b-4d14-83f9-428dbd56abfd
612023-12-30T06:23:53.707ZINFOcrucible-agent (worker): creating region Region { id: RegionId("7b95296d-4d6b-4d14-83f9-428dbd56abfd"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/7b95296d-4d6b-4d14-83f9-428dbd56abfd" region = 7b95296d-4d6b-4d14-83f9-428dbd56abfd
622023-12-30T06:23:53.883ZINFOcrucible-agent (worker): region files created ok region = 7b95296d-4d6b-4d14-83f9-428dbd56abfd
632023-12-30T06:23:53.883ZINFOcrucible-agent (datafile): region 7b95296d-4d6b-4d14-83f9-428dbd56abfd state: Requested -> Created
642023-12-30T06:23:53.883ZINFOcrucible-agent (worker): applying SMF actions post create...
652023-12-30T06:23:53.884ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-7b95296d-4d6b-4d14-83f9-428dbd56abfd
662023-12-30T06:23:53.891ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-7b95296d-4d6b-4d14-83f9-428dbd56abfd
672023-12-30T06:23:53.893ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:56260 req_id = aa6558ef-fdc8-455e-98b0-6c6393b5567b response_code = 200 uri = /crucible/0/regions
682023-12-30T06:23:53.903ZINFOcrucible-agent (worker): creating config property group
692023-12-30T06:23:53.906ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-7b95296d-4d6b-4d14-83f9-428dbd56abfd
702023-12-30T06:23:53.907ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/7b95296d-4d6b-4d14-83f9-428dbd56abfd
712023-12-30T06:23:53.907ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
722023-12-30T06:23:53.907ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
732023-12-30T06:23:53.907ZINFOcrucible-agent (worker): commit
742023-12-30T06:23:53.911ZINFOcrucible-agent (worker): ok!
752023-12-30T06:23:53.917ZINFOcrucible-agent (worker): SMF ok!
762023-12-30T06:23:54.316ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::b]:51664
772023-12-30T06:23:54.317ZINFOcrucible-agent (dropshot): request completed latency_us = 924 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:51664 req_id = 03061165-fed7-4ca3-b653-c1384e46ec71 response_code = 200 uri = /crucible/0/regions/7b95296d-4d6b-4d14-83f9-428dbd56abfd
782023-12-30T06:23:54.318ZINFOcrucible-agent (datafile): path is "/data/regions/7b95296d-4d6b-4d14-83f9-428dbd56abfd"
792023-12-30T06:23:54.327ZINFOcrucible-agent (datafile): dataset is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/7b95296d-4d6b-4d14-83f9-428dbd56abfd
802023-12-30T06:23:54.335ZINFOcrucible-agent (datafile): region 7b95296d-4d6b-4d14-83f9-428dbd56abfd state: Created -> Tombstoned
812023-12-30T06:23:54.336ZINFOcrucible-agent (worker): applying SMF actions before removal...
822023-12-30T06:23:54.336ZINFOcrucible-agent (dropshot): request completed latency_us = 18581 local_addr = [fd00:1122:3344:101::19]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:51664 req_id = ab42fd5b-91f3-4303-b715-91cbebed5197 response_code = 204 uri = /crucible/0/regions/7b95296d-4d6b-4d14-83f9-428dbd56abfd
832023-12-30T06:23:54.336ZINFOcrucible-agent (dropshot): request completed latency_us = 28 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:51664 req_id = 6f2fd0cf-e94c-4239-988d-8a45175f49b9 response_code = 200 uri = /crucible/0/regions/7b95296d-4d6b-4d14-83f9-428dbd56abfd
842023-12-30T06:23:54.337ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-7b95296d-4d6b-4d14-83f9-428dbd56abfd (instance states: (Some(Online), None))
852023-12-30T06:23:54.342ZINFOcrucible-agent (worker): SMF ok!
862023-12-30T06:23:54.351ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/7b95296d-4d6b-4d14-83f9-428dbd56abfd" region = 7b95296d-4d6b-4d14-83f9-428dbd56abfd
872023-12-30T06:23:54.402ZINFOcrucible-agent (datafile): region 7b95296d-4d6b-4d14-83f9-428dbd56abfd state: Tombstoned -> Destroyed
882023-12-30T06:23:54.439ZINFOcrucible-agent (dropshot): request completed latency_us = 102 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:51664 req_id = 132ffd40-bc33-46b4-b69d-3dfcfb36dfd4 response_code = 200 uri = /crucible/0/regions/7b95296d-4d6b-4d14-83f9-428dbd56abfd
892023-12-30T06:24:23.073ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:33645
902023-12-30T06:24:23.135ZINFOcrucible-agent (datafile): region 69c10c18-b360-467f-ad00-564c984bf576 state: Requested
912023-12-30T06:24:23.135ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
922023-12-30T06:24:23.135ZINFOcrucible-agent (dropshot): request completed latency_us = 273 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33645 req_id = b378e679-e586-4501-b255-1e83f272c4a4 response_code = 200 uri = /crucible/0/regions
932023-12-30T06:24:23.145ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/69c10c18-b360-467f-ad00-564c984bf576
942023-12-30T06:24:23.145ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/69c10c18-b360-467f-ad00-564c984bf576
952023-12-30T06:24:23.215ZINFOcrucible-agent (worker): creating region Region { id: RegionId("69c10c18-b360-467f-ad00-564c984bf576"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/69c10c18-b360-467f-ad00-564c984bf576" region = 69c10c18-b360-467f-ad00-564c984bf576
962023-12-30T06:24:23.354ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33645 req_id = 716880c7-38e0-448b-9858-84d7d9feb0e5 response_code = 200 uri = /crucible/0/regions
972023-12-30T06:24:23.354ZINFOcrucible-agent (worker): region files created ok region = 69c10c18-b360-467f-ad00-564c984bf576
982023-12-30T06:24:23.354ZINFOcrucible-agent (datafile): region 69c10c18-b360-467f-ad00-564c984bf576 state: Requested -> Created
992023-12-30T06:24:23.354ZINFOcrucible-agent (worker): applying SMF actions post create...
1002023-12-30T06:24:23.355ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-69c10c18-b360-467f-ad00-564c984bf576
1012023-12-30T06:24:23.362ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-69c10c18-b360-467f-ad00-564c984bf576
1022023-12-30T06:24:23.379ZINFOcrucible-agent (worker): creating config property group
1032023-12-30T06:24:23.388ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-69c10c18-b360-467f-ad00-564c984bf576
1042023-12-30T06:24:23.388ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/69c10c18-b360-467f-ad00-564c984bf576
1052023-12-30T06:24:23.388ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1062023-12-30T06:24:23.388ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
1072023-12-30T06:24:23.388ZINFOcrucible-agent (worker): commit
1082023-12-30T06:24:23.398ZINFOcrucible-agent (worker): ok!
1092023-12-30T06:24:23.411ZINFOcrucible-agent (worker): SMF ok!
1102023-12-30T06:24:23.861ZINFOcrucible-agent (dropshot): request completed latency_us = 817 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33645 req_id = 48838d67-7dbe-4205-bc53-09de8285d943 response_code = 200 uri = /crucible/0/regions
1112023-12-30T06:29:56.244ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:43702
1122023-12-30T06:29:56.244ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:43702 req_id = d6bc5c17-02df-473c-b256-98bd2c3d721c response_code = 200 uri = /crucible/0/regions/69c10c18-b360-467f-ad00-564c984bf576
1132023-12-30T06:29:56.246ZINFOcrucible-agent (datafile): path is "/data/regions/69c10c18-b360-467f-ad00-564c984bf576"
1142023-12-30T06:29:56.255ZINFOcrucible-agent (datafile): dataset is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/69c10c18-b360-467f-ad00-564c984bf576
1152023-12-30T06:29:56.264ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/69c10c18-b360-467f-ad00-564c984bf576@4bd36969-7ce7-4479-bc7f-aebaf430c842
1162023-12-30T06:29:56.264ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/69c10c18-b360-467f-ad00-564c984bf576", "4bd36969-7ce7-4479-bc7f-aebaf430c842"]
1172023-12-30T06:29:56.279ZINFOcrucible-agent (dropshot): request completed latency_us = 32391 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:43702 req_id = b05eec7c-2c04-4c4c-9efe-ea855fae385c response_code = 200 uri = /crucible/0/regions/69c10c18-b360-467f-ad00-564c984bf576/snapshots/4bd36969-7ce7-4479-bc7f-aebaf430c842
1182023-12-30T06:29:56.281ZINFOcrucible-agent (datafile): path is "/data/regions/69c10c18-b360-467f-ad00-564c984bf576"
1192023-12-30T06:29:56.289ZINFOcrucible-agent (datafile): dataset is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/69c10c18-b360-467f-ad00-564c984bf576
1202023-12-30T06:29:56.297ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/69c10c18-b360-467f-ad00-564c984bf576@4bd36969-7ce7-4479-bc7f-aebaf430c842
1212023-12-30T06:29:56.297ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/69c10c18-b360-467f-ad00-564c984bf576", "4bd36969-7ce7-4479-bc7f-aebaf430c842"]
1222023-12-30T06:29:56.306ZINFOcrucible-agent (datafile): requesting running snapshot 69c10c18-b360-467f-ad00-564c984bf576-4bd36969-7ce7-4479-bc7f-aebaf430c842 state: Requested
1232023-12-30T06:29:56.306ZINFOcrucible-agent (worker): applying SMF actions for region 69c10c18-b360-467f-ad00-564c984bf576 running snapshot 4bd36969-7ce7-4479-bc7f-aebaf430c842 (state Requested)...
1242023-12-30T06:29:56.306ZINFOcrucible-agent (dropshot): request completed latency_us = 25572 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43702 req_id = a9dbf415-e13f-420e-b886-0a35d391136f response_code = 200 uri = /crucible/0/regions/69c10c18-b360-467f-ad00-564c984bf576/snapshots/4bd36969-7ce7-4479-bc7f-aebaf430c842/run
1252023-12-30T06:29:56.315ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-69c10c18-b360-467f-ad00-564c984bf576-4bd36969-7ce7-4479-bc7f-aebaf430c842
1262023-12-30T06:29:56.318ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-69c10c18-b360-467f-ad00-564c984bf576-4bd36969-7ce7-4479-bc7f-aebaf430c842
1272023-12-30T06:29:56.332ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-69c10c18-b360-467f-ad00-564c984bf576-4bd36969-7ce7-4479-bc7f-aebaf430c842
1282023-12-30T06:29:56.332ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/69c10c18-b360-467f-ad00-564c984bf576/.zfs/snapshot/4bd36969-7ce7-4479-bc7f-aebaf430c842
1292023-12-30T06:29:56.332ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
1302023-12-30T06:29:56.332ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
1312023-12-30T06:29:56.332ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
1322023-12-30T06:29:56.332ZINFOcrucible-agent (worker): commit
1332023-12-30T06:29:56.337ZINFOcrucible-agent (worker): ok!
1342023-12-30T06:29:56.341ZINFOcrucible-agent (worker): SMF ok!
1352023-12-30T06:29:56.341ZINFOcrucible-agent (datafile): region 69c10c18-b360-467f-ad00-564c984bf576 running snapshot 4bd36969-7ce7-4479-bc7f-aebaf430c842 state: Requested -> Created