Max level shown:
1 [ Feb 7 22:41:13 Enabled. ]
2 [ Feb 7 22:41:13 Rereading configuration. ]
3 [ Feb 7 22:41:15 Rereading configuration. ]
4 [ Feb 7 22:41:17 Executing start method ("/opt/oxide/lib/svc/manifest/crucible/agent.sh"). ]
5 + . /lib/svc/share/smf_include.sh
6 ++ SMF_EXIT_OK=0
7 ++ SMF_EXIT_NODAEMON=94
8 ++ SMF_EXIT_ERR_FATAL=95
9 ++ SMF_EXIT_ERR_CONFIG=96
10 ++ SMF_EXIT_MON_DEGRADE=97
11 ++ SMF_EXIT_MON_OFFLINE=98
12 ++ SMF_EXIT_ERR_NOSMF=99
13 ++ SMF_EXIT_ERR_PERM=100
14 ++ svcprop -c -p config/datalink svc:/oxide/crucible/agent:default
15 + DATALINK=oxControlService12
16 ++ svcprop -c -p config/gateway svc:/oxide/crucible/agent:default
17 + GATEWAY=fd00:1122:3344:101::1
18 ++ svcprop -c -p config/dataset svc:/oxide/crucible/agent:default
19 + DATASET=oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible
20 ++ svcprop -c -p config/listen_addr svc:/oxide/crucible/agent:default
21 + LISTEN_ADDR=fd00:1122:3344:101::19
22 ++ svcprop -c -p config/listen_port svc:/oxide/crucible/agent:default
23 + LISTEN_PORT=32345
24 ++ svcprop -c -p config/portbase svc:/oxide/crucible/agent:default
25 + PORTBASE=19000
26 ++ svcprop -c -p config/downstairs_prefix svc:/oxide/crucible/agent:default
27 + DOWNSTAIRS_PREFIX=downstairs
28 ++ svcprop -c -p config/snapshot_prefix svc:/oxide/crucible/agent:default
29 + SNAPSHOT_PREFIX=snapshot
30 + [[ oxControlService12 == unknown ]]
31 + [[ fd00:1122:3344:101::1 == unknown ]]
32 + ipadm delete-if oxControlService12
33 ipadm: Could not delete oxControlService12: Interface does not exist
34 + true
35 + ipadm create-if -t oxControlService12
36 + ipadm set-ifprop -t -p mtu=9000 -m ipv4 oxControlService12
37 + ipadm set-ifprop -t -p mtu=9000 -m ipv6 oxControlService12
38 + ipadm show-addr oxControlService12/ll
39 ipadm: Address object not found
40 + ipadm create-addr -t -T addrconf oxControlService12/ll
41 + ipadm show-addr oxControlService12/omicron6
42 ipadm: Address object not found
43 + ipadm create-addr -t -T static -a fd00:1122:3344:101::19 oxControlService12/omicron6
44 + route get -inet6 default -inet6 fd00:1122:3344:101::1
45 default: not in table
46 + route add -inet6 default -inet6 fd00:1122:3344:101::1
47 add net default: gateway fd00:1122:3344:101::1
48 + args=('-D' '/opt/oxide/crucible/bin/crucible-downstairs' '--dataset' "$DATASET" '-l' "[$LISTEN_ADDR]:$LISTEN_PORT" '-P' "$PORTBASE" '-p' "$DOWNSTAIRS_PREFIX" '-s' "$SNAPSHOT_PREFIX")
49 + exec /opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible -l '[fd00:1122:3344:101::19]:32345' -P 19000 -p downstairs -s snapshot
50 note: configured to log to "/dev/stdout"
512024-02-07T22:41:18.545ZINFOcrucible-agent: dataset: "oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible"
522024-02-07T22:41:18.551ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::19]:32345
532024-02-07T22:41:18.551ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
542024-02-07T22:41:18.817ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::19]:32345
552024-02-07T22:42:06.304ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:57888
562024-02-07T22:42:06.365ZINFOcrucible-agent (datafile): region 40d9861c-fb1b-4ded-932f-aab2091ad717 state: Requested
572024-02-07T22:42:06.366ZINFOcrucible-agent (dropshot): request completed latency_us = 1855 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57888 req_id = c9350fb9-5957-4c38-ac01-1659258bd597 response_code = 200 uri = /crucible/0/regions
582024-02-07T22:42:06.366ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
592024-02-07T22:42:06.376ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/40d9861c-fb1b-4ded-932f-aab2091ad717
602024-02-07T22:42:06.376ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/40d9861c-fb1b-4ded-932f-aab2091ad717
612024-02-07T22:42:06.435ZINFOcrucible-agent (worker): creating region Region { id: RegionId("40d9861c-fb1b-4ded-932f-aab2091ad717"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/40d9861c-fb1b-4ded-932f-aab2091ad717" region = 40d9861c-fb1b-4ded-932f-aab2091ad717
622024-02-07T22:42:06.567ZINFOcrucible-agent (worker): region files created ok region = 40d9861c-fb1b-4ded-932f-aab2091ad717
632024-02-07T22:42:06.568ZINFOcrucible-agent (datafile): region 40d9861c-fb1b-4ded-932f-aab2091ad717 state: Requested -> Created
642024-02-07T22:42:06.568ZINFOcrucible-agent (worker): applying SMF actions post create...
652024-02-07T22:42:06.568ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-40d9861c-fb1b-4ded-932f-aab2091ad717
662024-02-07T22:42:06.571ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-40d9861c-fb1b-4ded-932f-aab2091ad717
672024-02-07T22:42:06.579ZINFOcrucible-agent (worker): creating config property group
682024-02-07T22:42:06.582ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-40d9861c-fb1b-4ded-932f-aab2091ad717
692024-02-07T22:42:06.583ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/40d9861c-fb1b-4ded-932f-aab2091ad717
702024-02-07T22:42:06.583ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
712024-02-07T22:42:06.583ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
722024-02-07T22:42:06.583ZINFOcrucible-agent (worker): commit
732024-02-07T22:42:06.587ZINFOcrucible-agent (worker): ok!
742024-02-07T22:42:06.591ZINFOcrucible-agent (worker): SMF ok!
752024-02-07T22:42:06.651ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57888 req_id = 7b4be511-4202-4f12-ba04-45eeace169b9 response_code = 200 uri = /crucible/0/regions
762024-02-07T22:42:07.382ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:60757
772024-02-07T22:42:07.382ZINFOcrucible-agent (dropshot): request completed latency_us = 105 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:60757 req_id = 64adc142-14ee-41da-89d6-4afb962a5af8 response_code = 200 uri = /crucible/0/regions/40d9861c-fb1b-4ded-932f-aab2091ad717
782024-02-07T22:42:07.383ZINFOcrucible-agent (datafile): path is "/data/regions/40d9861c-fb1b-4ded-932f-aab2091ad717"
792024-02-07T22:42:07.392ZINFOcrucible-agent (datafile): dataset is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/40d9861c-fb1b-4ded-932f-aab2091ad717
802024-02-07T22:42:07.399ZINFOcrucible-agent (datafile): region 40d9861c-fb1b-4ded-932f-aab2091ad717 state: Created -> Tombstoned
812024-02-07T22:42:07.399ZINFOcrucible-agent (worker): applying SMF actions before removal...
822024-02-07T22:42:07.399ZINFOcrucible-agent (dropshot): request completed latency_us = 16589 local_addr = [fd00:1122:3344:101::19]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:60757 req_id = fcd6597d-71ed-42af-b4ca-2a86397826d3 response_code = 204 uri = /crucible/0/regions/40d9861c-fb1b-4ded-932f-aab2091ad717
832024-02-07T22:42:07.399ZINFOcrucible-agent (dropshot): request completed latency_us = 38 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:60757 req_id = f0ec4529-f3d1-4875-a64f-85cd72b20139 response_code = 200 uri = /crucible/0/regions/40d9861c-fb1b-4ded-932f-aab2091ad717
842024-02-07T22:42:07.400ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-40d9861c-fb1b-4ded-932f-aab2091ad717 (instance states: (Some(Online), None))
852024-02-07T22:42:07.404ZINFOcrucible-agent (worker): SMF ok!
862024-02-07T22:42:07.412ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/40d9861c-fb1b-4ded-932f-aab2091ad717" region = 40d9861c-fb1b-4ded-932f-aab2091ad717
872024-02-07T22:42:07.464ZINFOcrucible-agent (datafile): region 40d9861c-fb1b-4ded-932f-aab2091ad717 state: Tombstoned -> Destroyed
882024-02-07T22:42:07.515ZINFOcrucible-agent (dropshot): request completed latency_us = 82 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:60757 req_id = 9776c63c-47e7-4041-8ced-ad2615407314 response_code = 200 uri = /crucible/0/regions/40d9861c-fb1b-4ded-932f-aab2091ad717
892024-02-07T22:42:08.599ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:51212
902024-02-07T22:42:08.629ZINFOcrucible-agent (datafile): region 62d7d11f-b007-432c-a631-2d570c318ef1 state: Requested
912024-02-07T22:42:08.629ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
922024-02-07T22:42:08.629ZINFOcrucible-agent (dropshot): request completed latency_us = 855 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51212 req_id = 5db081e9-af98-44ae-810a-c8ddd99936d4 response_code = 200 uri = /crucible/0/regions
932024-02-07T22:42:08.638ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/62d7d11f-b007-432c-a631-2d570c318ef1
942024-02-07T22:42:08.638ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/62d7d11f-b007-432c-a631-2d570c318ef1
952024-02-07T22:42:08.690ZINFOcrucible-agent (worker): creating region Region { id: RegionId("62d7d11f-b007-432c-a631-2d570c318ef1"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/62d7d11f-b007-432c-a631-2d570c318ef1" region = 62d7d11f-b007-432c-a631-2d570c318ef1
962024-02-07T22:42:08.765ZINFOcrucible-agent (worker): region files created ok region = 62d7d11f-b007-432c-a631-2d570c318ef1
972024-02-07T22:42:08.765ZINFOcrucible-agent (datafile): region 62d7d11f-b007-432c-a631-2d570c318ef1 state: Requested -> Created
982024-02-07T22:42:08.766ZINFOcrucible-agent (worker): applying SMF actions post create...
992024-02-07T22:42:08.767ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-62d7d11f-b007-432c-a631-2d570c318ef1
1002024-02-07T22:42:08.770ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-62d7d11f-b007-432c-a631-2d570c318ef1
1012024-02-07T22:42:08.781ZINFOcrucible-agent (worker): creating config property group
1022024-02-07T22:42:08.785ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-62d7d11f-b007-432c-a631-2d570c318ef1
1032024-02-07T22:42:08.785ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/62d7d11f-b007-432c-a631-2d570c318ef1
1042024-02-07T22:42:08.785ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1052024-02-07T22:42:08.785ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
1062024-02-07T22:42:08.785ZINFOcrucible-agent (worker): commit
1072024-02-07T22:42:08.789ZINFOcrucible-agent (worker): ok!
1082024-02-07T22:42:08.794ZINFOcrucible-agent (worker): SMF ok!
1092024-02-07T22:42:08.987ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51212 req_id = fc1f33e5-c2ec-44f5-8307-55085d3f591e response_code = 200 uri = /crucible/0/regions