after upgrading from

2.0.3

to

2.1.0

, all my existing access tokens stopped working, debug log is showing this:

{"audience":"application","error":{"debug":"not_found","message":"request_unauthorized","reason":"Check that you provided valid credentials in the right format.","stack_trace":"\<http://ngithub.com/ory/x/errorsx.WithStack|ngithub.com/ory/x/errorsx.WithStack>\n\t/go/pkg/mod/github.com/ory/[email protected]/errorsx/errors.go:41\ngithub.com/ory/hydra/v2/persistence/sql.(*Persister).findSessionBySignature.func1\n\t/project/persistence/sql/persister_oauth2.go:265\ngithub.com/ory/x/popx.Transaction.func2\n\t/go/pkg/mod/github.com/ory/[email protected]/popx/transaction.go:45\ngithub.com/gobuffalo/pop/v6.(*Connection).Transaction.func1\n\t/go/pkg/mod/github.com/gobuffalo/pop/[email protected]/connection.go:176\ngithub.com/gobuffalo/pop/v6.commonDialect.Lock\n\t/go/pkg/mod/github.com/gobuffalo/pop/[email protected]/dialect_common.go:29\ngithub.com/gobuffalo/pop/v6.(*Connection).Transaction\n\t/go/pkg/mod/github.com/gobuffalo/pop/[email protected]/connection.go:156\ngithub.com/ory/x/popx.Transaction\n\t/go/pkg/mod/github.com/ory/[email protected]/popx/transaction.go:44\ngithub.com/ory/hydra/v2/persistence/sql.(*Persister).transaction\n\t/project/persistence/sql/persister.go:181\ngithub.com/ory/hydra/v2/persistence/sql.(*Persister).findSessionBySignature\n\t/project/persistence/sql/persister_oauth2.go:256\ngithub.com/ory/hydra/v2/persistence/sql.(*Persister).GetAccessTokenSession\n\t/project/persistence/sql/persister_oauth2.go:382\ngithub.com/ory/fosite/handler/oauth2.(*CoreValidator).introspectAccessToken\n\t/go/pkg/mod/github.com/ory/[email protected]/handler/oauth2/introspector.go:71\ngithub.com/ory/fosite/handler/oauth2.(*CoreValidator).IntrospectToken\n\t/go/pkg/mod/github.com/ory/[email protected]/handler/oauth2/introspector.go:46\ngithub.com/ory/fosite.(*Fosite).IntrospectToken\n\t/go/pkg/mod/github.com/ory/[email protected]/introspect.go:45\ngithub.com/ory/hydra/v2/oauth2.(*Handler).introspectOAuth2Token\n\t/project/oauth2/handler.go:734\ngithub.com/ory/x/httprouterx.NoCacheHandle.func1\n\t/go/pkg/mod/github.com/ory/[email protected]/httprouterx/nocache.go:21\ngithub.com/julienschmidt/httprouter.(*Router).ServeHTTP\n\t/go/pkg/mod/github.com/julienschmidt/[email protected]/router.go:387\ngithub.com/urfave/negroni.Wrap.func1\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:46\ngithub.com/urfave/negroni.HandlerFunc.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:29\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2109\ngithub.com/ory/hydra/v2/x.RejectInsecureRequests.func1\n\t/project/x/tls_termination.go:65\ngithub.com/urfave/negroni.HandlerFunc.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:29\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\ngithub.com/ory/x/metricsx.(*Service).ServeHTTP\n\t/go/pkg/mod/github.com/ory/[email protected]/metricsx/middleware.go:258\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2109\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1\n\t/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:284\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2109\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1\n\t/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:142\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2109\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func1\n\t/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:92\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2109","status":"Unauthorized","status_code":401},"http_request":{"headers":{"accept-encoding":"gzip","content-length":"93","content-type":"application/x-www-form-urlencoded","user-agent":"Go-http-client/1.1","x-b3-parentspanid":"9cc4d93e5ee209f5","x-b3-sampled":"0","x-b3-spanid":"0c43246f38db3c97","x-b3-traceid":"d74c010831b4dbc29cc4d93e5ee209f5","x-envoy-attempt-count":"1","x-forwarded-client-cert":"By=<spiffe://cluster.local/ns/idp/sa/default;Hash=e6b987bee4008c346fa7b4b6b35b9f6387949c2af2ca891912afeab83222e55a;Subject=>\"\";URI=<spiffe://cluster.local/ns/idp/sa/default>","x-forwarded-proto":"https","x-request-id":"44253d08-bc98-45a9-a56f-426fd009c6ea"},"host":"hydra-admin.idp.svc.cluster.local","method":"POST","path":"/admin/oauth2/introspect","query":null,"remote":"127.0.0.6:48105","scheme":"http"},"level":"info","msg":"access denied","otel":{"span_id":"517260162c0fd4b3","trace_id":"a1732b8d06c5ec1e6de13b3d37b12b5b"},"service_name":"Ory Hydra","service_version":"v2.1.0","time":"2023-04-10T12:52:28.70243505Z"}

I've ran the migration and all worked fine. before that I had similar issues when upgrading from

1.9.x

to

2.0.x

, access tokens stopped working and I had to re-issue them. I'm curious if I'm missing some config and whether it has to anything to do with secret rotation.