I'm laid off and been called back to work. I'm not wanting to go back, is there a possibility that they can make me pay back my unemployment benifits?

I'm laid off and been called back to work. I'm not wanting to go back, is there a possibility that they can make me pay back my unemployment benifits?