SSC 8.8 Authentication failed: no Authorization header

I recently upgraded my LCM deployed SSC server to 8.8.x. If you need a guide to go through the upgrade you can find my other post here.

After the upgrade was completed i was noticing strange behavior in the SSC UI so i checked the status the of the services. Here are the errors i found and how i fixed them

The first step was to check the status of the service

systemctl status salt-master

The return was this

* salt-master.service - The Salt Master Server
   Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2022-07-16 20:30:29 UTC; 1 day 2h ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltproject.io/en/latest/contents.html
 Main PID: 801 (salt-master)
    Tasks: 40 (limit: 9830)
   Memory: 499.5M
   CGroup: /system.slice/salt-master.service
           |-  801 /bin/python3 /usr/bin/salt-master
           |- 1005 /bin/python3 /usr/bin/salt-master
           |- 1088 /bin/python3 /usr/bin/salt-master
           |- 1090 /bin/python3 /usr/bin/salt-master
           |- 1101 /bin/python3 /usr/bin/salt-master
           |- 1102 /bin/python3 /usr/bin/salt-master
           |- 1110 /bin/python3 /usr/bin/salt-master
           |- 1113 /bin/python3 /usr/bin/salt-master
           |- 1119 /bin/python3 /usr/bin/salt-master
           |- 1120 /bin/python3 /usr/bin/salt-master
           |- 1397 /bin/python3 /usr/bin/salt-master
           |- 1398 /bin/python3 /usr/bin/salt-master
           |- 1400 /bin/python3 /usr/bin/salt-master
           |- 1410 /bin/python3 /usr/bin/salt-master
           |- 1414 /bin/python3 /usr/bin/salt-master
           |- 1419 /bin/python3 /usr/bin/salt-master
           |- 1420 /bin/python3 /usr/bin/salt-master
           |- 1424 /bin/python3 /usr/bin/salt-master
           `-15430 /bin/python3 /usr/bin/salt-master

Jul 17 21:07:47 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to authenticate: Authentication failed: no Authorization header
Jul 17 21:07:47 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to send minion key state to SSE: 401 Authentication failed: no Authorization header
Jul 17 21:07:48 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to authenticate: Authentication failed: no Authorization header
Jul 17 21:07:48 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to send minion cache to SSE: 401 Authentication failed: no Authorization header
Jul 17 21:07:48 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to authenticate: Authentication failed: no Authorization header
Jul 17 21:07:48 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to send master fileserver data to SSE: 401 Authentication failed: no Authorization header
Jul 17 21:07:50 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to authenticate: Authentication failed: no Authorization header
Jul 17 21:07:50 ssc-01a.corp.local salt-master[801]: [ERROR   ] sseapi_event_queue: failed to send entries to SSE (will requeue): 401 Authentication failed: no Authorization header
Jul 17 21:07:55 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to authenticate: Authentication failed: no Authorization header
Jul 17 21:07:55 ssc-01a.corp.local salt-master[801]: [ERROR   ] sseapi_event_queue: failed to send entries to SSE (will requeue): 401 Authentication failed: no Authorization header

The first step to resolve the error was to delete the master key from the UI by going to SSC UI -> Administration -> Master Keys -> Accepted -> Select the old key and click on delete ex:

Next we need to stop the salt master service by running

systemctl stop salt-master

Additionally on the cli we also need to delete the old key file located at:

/etc/salt/pki/master/sseapi_key.pub

We can delete it by running:

rm /etc/salt/pki/master/sseapi_key.pub

Once the above steps are complete we can get start the services again and accept the new key in the UI.

We can start the service back up by running:

systemctl start salt-master

We can now check the service and add the key back in the UI

systemctl status salt-master

Finally we can restart the saltstack service and verify that its running without errors:

* salt-master.service - The Salt Master Server
   Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2022-05-15 20:02:56 UTC; 51s ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltproject.io/en/latest/contents.html
 Main PID: 31309 (salt-master)
    Tasks: 39 (limit: 9830)
   Memory: 330.0M
   CGroup: /system.slice/salt-master.service
           |-31309 /bin/python3 /usr/bin/salt-master
           |-31315 /bin/python3 /usr/bin/salt-master
           |-31320 /bin/python3 /usr/bin/salt-master
           |-31323 /bin/python3 /usr/bin/salt-master
           |-31325 /bin/python3 /usr/bin/salt-master
           |-31326 /bin/python3 /usr/bin/salt-master
           |-31327 /bin/python3 /usr/bin/salt-master
           |-31328 /bin/python3 /usr/bin/salt-master
           |-31330 /bin/python3 /usr/bin/salt-master
           |-31397 /bin/python3 /usr/bin/salt-master
           |-31398 /bin/python3 /usr/bin/salt-master
           |-31400 /bin/python3 /usr/bin/salt-master
           |-31411 /bin/python3 /usr/bin/salt-master
           |-31412 /bin/python3 /usr/bin/salt-master
           |-31413 /bin/python3 /usr/bin/salt-master
           |-31414 /bin/python3 /usr/bin/salt-master
           |-31415 /bin/python3 /usr/bin/salt-master
           `-31416 /bin/python3 /usr/bin/salt-master

May 15 20:02:54 ssc-01a.corp.local systemd[1]: Starting The Salt Master Server...
May 15 20:02:56 ssc-01a.corp.local systemd[1]: Started The Salt Master Server.

If the status page a return similar to this

* salt-master.service - The Salt Master Server
   Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2022-07-17 23:19:35 UTC; 3min 24s ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltproject.io/en/latest/contents.html
 Main PID: 21532 (salt-master)
    Tasks: 40 (limit: 9830)
   Memory: 346.7M
   CGroup: /system.slice/salt-master.service
           |-21532 /bin/python3 /usr/bin/salt-master
           |-21538 /bin/python3 /usr/bin/salt-master
           |-21546 /bin/python3 /usr/bin/salt-master
           |-21550 /bin/python3 /usr/bin/salt-master
           |-21552 /bin/python3 /usr/bin/salt-master
           |-21553 /bin/python3 /usr/bin/salt-master
           |-21554 /bin/python3 /usr/bin/salt-master
           |-21555 /bin/python3 /usr/bin/salt-master
           |-21556 /bin/python3 /usr/bin/salt-master
           |-21557 /bin/python3 /usr/bin/salt-master
           |-21628 /bin/python3 /usr/bin/salt-master
           |-21629 /bin/python3 /usr/bin/salt-master
           |-21631 /bin/python3 /usr/bin/salt-master
           |-21641 /bin/python3 /usr/bin/salt-master
           |-21644 /bin/python3 /usr/bin/salt-master
           |-21645 /bin/python3 /usr/bin/salt-master
           |-21646 /bin/python3 /usr/bin/salt-master
           |-21647 /bin/python3 /usr/bin/salt-master
           `-21648 /bin/python3 /usr/bin/salt-master

Jul 17 23:19:33 ssc-01a.corp.local systemd[1]: Starting The Salt Master Server...
Jul 17 23:19:35 ssc-01a.corp.local systemd[1]: Started The Salt Master Server.

Next we need to go back to the UI and accept the new master key. SSC UI -> Administration -> Master Keys -> Pending Select the new key and click on Accept Key