Max level shown:
1 [ Dec 11 18:48:06 Enabled. ]
2 [ Dec 11 18:48:06 Rereading configuration. ]
3 [ Dec 11 18:48:11 Rereading configuration. ]
4 [ Dec 11 18:48:19 Executing start method ("/opt/oxide/lib/svc/manifest/crucible/agent.sh"). ]
5 + . /lib/svc/share/smf_include.sh
6 ++ SMF_EXIT_OK=0
7 ++ SMF_EXIT_NODAEMON=94
8 ++ SMF_EXIT_ERR_FATAL=95
9 ++ SMF_EXIT_ERR_CONFIG=96
10 ++ SMF_EXIT_MON_DEGRADE=97
11 ++ SMF_EXIT_MON_OFFLINE=98
12 ++ SMF_EXIT_ERR_NOSMF=99
13 ++ SMF_EXIT_ERR_PERM=100
14 ++ svcprop -c -p config/datalink svc:/oxide/crucible/agent:default
15 + DATALINK=oxControlService22
16 ++ svcprop -c -p config/gateway svc:/oxide/crucible/agent:default
17 + GATEWAY=fd00:1122:3344:101::1
18 ++ svcprop -c -p config/dataset svc:/oxide/crucible/agent:default
19 + DATASET=oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible
20 ++ svcprop -c -p config/listen_addr svc:/oxide/crucible/agent:default
21 + LISTEN_ADDR=fd00:1122:3344:101::12
22 ++ svcprop -c -p config/listen_port svc:/oxide/crucible/agent:default
23 + LISTEN_PORT=32345
24 ++ svcprop -c -p config/portbase svc:/oxide/crucible/agent:default
25 + PORTBASE=19000
26 ++ svcprop -c -p config/downstairs_prefix svc:/oxide/crucible/agent:default
27 + DOWNSTAIRS_PREFIX=downstairs
28 ++ svcprop -c -p config/snapshot_prefix svc:/oxide/crucible/agent:default
29 + SNAPSHOT_PREFIX=snapshot
30 + [[ oxControlService22 == unknown ]]
31 + [[ fd00:1122:3344:101::1 == unknown ]]
32 + ipadm delete-if oxControlService22
33 ipadm: Could not delete oxControlService22: Interface does not exist
34 + true
35 + ipadm create-if -t oxControlService22
36 + ipadm set-ifprop -t -p mtu=9000 -m ipv4 oxControlService22
37 + ipadm set-ifprop -t -p mtu=9000 -m ipv6 oxControlService22
38 + ipadm show-addr oxControlService22/ll
39 ipadm: Address object not found
40 + ipadm create-addr -t -T addrconf oxControlService22/ll
41 + ipadm show-addr oxControlService22/omicron6
42 ipadm: Address object not found
43 + ipadm create-addr -t -T static -a fd00:1122:3344:101::12 oxControlService22/omicron6
44 + route get -inet6 default -inet6 fd00:1122:3344:101::1
45 default: not in table
46 + route add -inet6 default -inet6 fd00:1122:3344:101::1
47 add net default: gateway fd00:1122:3344:101::1
48 + args=('-D' '/opt/oxide/crucible/bin/crucible-downstairs' '--dataset' "$DATASET" '-l' "[$LISTEN_ADDR]:$LISTEN_PORT" '-P' "$PORTBASE" '-p' "$DOWNSTAIRS_PREFIX" '-s' "$SNAPSHOT_PREFIX")
49 + exec /opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible -l '[fd00:1122:3344:101::12]:32345' -P 19000 -p downstairs -s snapshot
50 note: configured to log to "/dev/stdout"
512023-12-11T18:48:20.154ZINFOcrucible-agent: dataset: "oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible"
522023-12-11T18:48:20.158ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::12]:32345
532023-12-11T18:48:20.158ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
542023-12-11T18:48:20.517ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::12]:32345
552023-12-11T18:49:33.410ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:50253
562023-12-11T18:49:33.443ZINFOcrucible-agent (datafile): region b0a3780e-689f-4ea9-bcbd-af5e5de60841 state: Requested
572023-12-11T18:49:33.443ZINFOcrucible-agent (dropshot): request completed latency_us = 13057 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50253 req_id = 839438a1-3e5a-4856-8134-591da4c13c8c response_code = 200 uri = /crucible/0/regions
582023-12-11T18:49:33.444ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
592023-12-11T18:49:33.454ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841
602023-12-11T18:49:33.454ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841
612023-12-11T18:49:33.503ZINFOcrucible-agent (worker): creating region Region { id: RegionId("b0a3780e-689f-4ea9-bcbd-af5e5de60841"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841" region = b0a3780e-689f-4ea9-bcbd-af5e5de60841
622023-12-11T18:49:33.668ZINFOcrucible-agent (worker): region files created ok region = b0a3780e-689f-4ea9-bcbd-af5e5de60841
632023-12-11T18:49:33.668ZINFOcrucible-agent (datafile): region b0a3780e-689f-4ea9-bcbd-af5e5de60841 state: Requested -> Created
642023-12-11T18:49:33.668ZINFOcrucible-agent (worker): applying SMF actions post create...
652023-12-11T18:49:33.673ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-b0a3780e-689f-4ea9-bcbd-af5e5de60841
662023-12-11T18:49:33.679ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-b0a3780e-689f-4ea9-bcbd-af5e5de60841
672023-12-11T18:49:33.696ZINFOcrucible-agent (worker): creating config property group
682023-12-11T18:49:33.699ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-b0a3780e-689f-4ea9-bcbd-af5e5de60841
692023-12-11T18:49:33.700ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841
702023-12-11T18:49:33.700ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
712023-12-11T18:49:33.700ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
722023-12-11T18:49:33.700ZINFOcrucible-agent (worker): commit
732023-12-11T18:49:33.704ZINFOcrucible-agent (worker): ok!
742023-12-11T18:49:33.717ZINFOcrucible-agent (worker): SMF ok!
752023-12-11T18:49:33.782ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50253 req_id = 174b9b52-60a2-49a8-8201-ad8816a3e001 response_code = 200 uri = /crucible/0/regions
762023-12-11T18:54:58.986ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:44879
772023-12-11T18:54:58.990ZINFOcrucible-agent (dropshot): request completed latency_us = 3903 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:44879 req_id = 0ec661df-38de-4353-8e37-50b6743ac183 response_code = 200 uri = /crucible/0/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841
782023-12-11T18:54:58.992ZINFOcrucible-agent (datafile): path is "/data/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841"
792023-12-11T18:54:59.001ZINFOcrucible-agent (datafile): dataset is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841
802023-12-11T18:54:59.010ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841@10d2c586-80aa-4e97-ab5c-713c478e9e4c
812023-12-11T18:54:59.010ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841", "10d2c586-80aa-4e97-ab5c-713c478e9e4c"]
822023-12-11T18:54:59.024ZINFOcrucible-agent (dropshot): request completed latency_us = 31333 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:44879 req_id = cd3137fe-f415-4669-a1e9-ed42a16dbfe6 response_code = 200 uri = /crucible/0/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841/snapshots/10d2c586-80aa-4e97-ab5c-713c478e9e4c
832023-12-11T18:54:59.026ZINFOcrucible-agent (datafile): path is "/data/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841"
842023-12-11T18:54:59.035ZINFOcrucible-agent (datafile): dataset is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841
852023-12-11T18:54:59.044ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841@10d2c586-80aa-4e97-ab5c-713c478e9e4c
862023-12-11T18:54:59.044ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841", "10d2c586-80aa-4e97-ab5c-713c478e9e4c"]
872023-12-11T18:54:59.053ZINFOcrucible-agent (datafile): requesting running snapshot b0a3780e-689f-4ea9-bcbd-af5e5de60841-10d2c586-80aa-4e97-ab5c-713c478e9e4c state: Requested
882023-12-11T18:54:59.053ZINFOcrucible-agent (worker): applying SMF actions for region b0a3780e-689f-4ea9-bcbd-af5e5de60841 running snapshot 10d2c586-80aa-4e97-ab5c-713c478e9e4c (state Requested)...
892023-12-11T18:54:59.053ZINFOcrucible-agent (dropshot): request completed latency_us = 26844 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:44879 req_id = fee14d2a-9a78-4808-bbe3-75c0d10e03f2 response_code = 200 uri = /crucible/0/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841/snapshots/10d2c586-80aa-4e97-ab5c-713c478e9e4c/run
902023-12-11T18:54:59.062ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-b0a3780e-689f-4ea9-bcbd-af5e5de60841-10d2c586-80aa-4e97-ab5c-713c478e9e4c
912023-12-11T18:54:59.066ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-b0a3780e-689f-4ea9-bcbd-af5e5de60841-10d2c586-80aa-4e97-ab5c-713c478e9e4c
922023-12-11T18:54:59.087ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-b0a3780e-689f-4ea9-bcbd-af5e5de60841-10d2c586-80aa-4e97-ab5c-713c478e9e4c
932023-12-11T18:54:59.088ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b0a3780e-689f-4ea9-bcbd-af5e5de60841/.zfs/snapshot/10d2c586-80aa-4e97-ab5c-713c478e9e4c
942023-12-11T18:54:59.088ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
952023-12-11T18:54:59.088ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
962023-12-11T18:54:59.088ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
972023-12-11T18:54:59.088ZINFOcrucible-agent (worker): commit
982023-12-11T18:54:59.095ZINFOcrucible-agent (worker): ok!
992023-12-11T18:54:59.102ZINFOcrucible-agent (worker): SMF ok!
1002023-12-11T18:54:59.102ZINFOcrucible-agent (datafile): region b0a3780e-689f-4ea9-bcbd-af5e5de60841 running snapshot 10d2c586-80aa-4e97-ab5c-713c478e9e4c state: Requested -> Created
1012023-12-11T18:55:01.852ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:43507
1022023-12-11T18:55:01.882ZINFOcrucible-agent (datafile): region 972c4021-f3ab-44db-b335-c26e403b7e27 state: Requested
1032023-12-11T18:55:01.885ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
1042023-12-11T18:55:01.885ZINFOcrucible-agent (dropshot): request completed latency_us = 3187 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43507 req_id = 82b06bfb-8017-439e-8a1b-373c7266e6e1 response_code = 200 uri = /crucible/0/regions
1052023-12-11T18:55:01.893ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/972c4021-f3ab-44db-b335-c26e403b7e27
1062023-12-11T18:55:01.893ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/972c4021-f3ab-44db-b335-c26e403b7e27
1072023-12-11T18:55:01.970ZINFOcrucible-agent (worker): creating region Region { id: RegionId("972c4021-f3ab-44db-b335-c26e403b7e27"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/972c4021-f3ab-44db-b335-c26e403b7e27" region = 972c4021-f3ab-44db-b335-c26e403b7e27
1082023-12-11T18:55:02.039ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43507 req_id = 3399fe09-3f01-412d-b251-1a2f695b980a response_code = 200 uri = /crucible/0/regions
1092023-12-11T18:55:02.080ZINFOcrucible-agent (worker): region files created ok region = 972c4021-f3ab-44db-b335-c26e403b7e27
1102023-12-11T18:55:02.080ZINFOcrucible-agent (datafile): region 972c4021-f3ab-44db-b335-c26e403b7e27 state: Requested -> Created
1112023-12-11T18:55:02.080ZINFOcrucible-agent (worker): applying SMF actions post create...
1122023-12-11T18:55:02.081ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-972c4021-f3ab-44db-b335-c26e403b7e27
1132023-12-11T18:55:02.084ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-972c4021-f3ab-44db-b335-c26e403b7e27
1142023-12-11T18:55:02.097ZINFOcrucible-agent (worker): creating config property group
1152023-12-11T18:55:02.105ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-972c4021-f3ab-44db-b335-c26e403b7e27
1162023-12-11T18:55:02.105ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/972c4021-f3ab-44db-b335-c26e403b7e27
1172023-12-11T18:55:02.105ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1182023-12-11T18:55:02.105ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
1192023-12-11T18:55:02.105ZINFOcrucible-agent (worker): commit
1202023-12-11T18:55:02.115ZINFOcrucible-agent (worker): ok!
1212023-12-11T18:55:02.130ZINFOcrucible-agent (worker): SMF ok!
1222023-12-11T18:55:02.568ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43507 req_id = 0248541e-a831-4608-834e-cd439343ea73 response_code = 200 uri = /crucible/0/regions
1232023-12-11T18:55:32.833ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:41215
1242023-12-11T18:55:32.833ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:41215 req_id = bf13267f-3f44-43ae-bdec-f8a96d2e4919 response_code = 200 uri = /crucible/0/regions/972c4021-f3ab-44db-b335-c26e403b7e27
1252023-12-11T18:55:32.834ZINFOcrucible-agent (datafile): path is "/data/regions/972c4021-f3ab-44db-b335-c26e403b7e27"
1262023-12-11T18:55:32.848ZINFOcrucible-agent (datafile): dataset is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/972c4021-f3ab-44db-b335-c26e403b7e27
1272023-12-11T18:55:32.858ZINFOcrucible-agent (datafile): region 972c4021-f3ab-44db-b335-c26e403b7e27 state: Created -> Tombstoned
1282023-12-11T18:55:32.858ZINFOcrucible-agent (worker): applying SMF actions before removal...
1292023-12-11T18:55:32.859ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-972c4021-f3ab-44db-b335-c26e403b7e27 (instance states: (Some(Online), None))
1302023-12-11T18:55:32.861ZINFOcrucible-agent (dropshot): request completed latency_us = 27013 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:41215 req_id = 39996d39-0489-4db0-8686-fe4e9f746bf5 response_code = 204 uri = /crucible/0/regions/972c4021-f3ab-44db-b335-c26e403b7e27
1312023-12-11T18:55:32.861ZINFOcrucible-agent (dropshot): request completed latency_us = 35 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:41215 req_id = e8177c6f-971b-4a19-9cc2-fb58407b1118 response_code = 200 uri = /crucible/0/regions/972c4021-f3ab-44db-b335-c26e403b7e27
1322023-12-11T18:55:32.867ZINFOcrucible-agent (worker): SMF ok!
1332023-12-11T18:55:32.877ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/972c4021-f3ab-44db-b335-c26e403b7e27" region = 972c4021-f3ab-44db-b335-c26e403b7e27
1342023-12-11T18:55:32.915ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:41215 req_id = ba5f0955-ef92-4cd2-8e35-0a8a00d14c26 response_code = 200 uri = /crucible/0/regions/972c4021-f3ab-44db-b335-c26e403b7e27
1352023-12-11T18:55:32.926ZINFOcrucible-agent (datafile): region 972c4021-f3ab-44db-b335-c26e403b7e27 state: Tombstoned -> Destroyed
1362023-12-11T18:55:33.048ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:41215 req_id = 91265bc1-37a0-464d-921c-52f2ce32cbba response_code = 200 uri = /crucible/0/regions/972c4021-f3ab-44db-b335-c26e403b7e27