Implementing NextAuth with custom OAuth for Next.js frontend and Java backend

Frontend
1

I’m trying to set up NextAuth with a custom OAuth provider. My setup is:

  • Next.js frontend
  • Java backend
  • Third-party OAuth server

I’m not sure how to make this work. Should I:

  1. Send users to the OAuth server from the frontend?
  2. Handle the OAuth flow in the backend?

I’ve tried both ways but got stuck. With the frontend approach, I’m not sure how to construct the OAuth URL or handle the callback. With the backend approach, I’m confused about how to pass the tokens back to the frontend after redirects.

The main issues I’m facing:

  • Keeping the flow secure
  • Properly using NextAuth callbacks
  • Managing redirects and data flow

I’ve seen some solutions online, but they often mix Next.js into the backend, which I don’t want. I need the Java backend to handle security.

Has anyone successfully implemented this kind of setup? Any tips on the best approach or common pitfalls to avoid?

// Example of what I'm trying to do
const handleLogin = async () => {
  // Not sure if this is the right approach
  const result = await signIn('custom-provider')
  // How do I handle the result and set up the session?
}

Any help would be appreciated!

2

i’ve dealt with this before. my advice: keep oauth on the backend. it’s safer that way. you can make an api endpoint in java to start the flow, then handle the callback there too. for the frontend, just redirect to that endpoint when the user wants to log in. nextauth can work with custom sessions, so you should be able to integrate it once you’ve got the backend sorted. good luck!

3

Having implemented a similar setup, I recommend handling the OAuth flow primarily on the backend to ensure better security and control. In my experience, initiating the login through a Java backend endpoint lets you securely construct the OAuth URL and manage redirection. Once the OAuth server completes authentication, it should callback to your backend, which can exchange the authorization code for tokens and establish a session. Employing secure HTTP-only cookies helps protect session data. Customizing NextAuth with your session management system is achievable with a careful configuration of callbacks.

4

hey there! have you considered using a hybrid approach? maybe start the flow on the frontend, but let the backend handle the heavy lifting? i’m curious how you’re managing state between redirects. what security measures are you taking to protect against CSRF attacks? it’d be cool to hear more about your specific OAuth implementation!