The system call setfsgid sets the group ID that the Linux kernel uses to check for all accesses to the file system. Normally, the value of fsgid will shadow the value of the effective group ID. In fact, whenever the effective group ID is changed, fsgid will also be changed to the new value of the effective group ID.
Explicit calls to setfsuid and setfsgid are usually only used by programs such as the Linux NFS server that need to change what user and group ID is used for file access without a corresponding change in the real and effective user and group IDs. A change in the normal user IDs for a program such as the NFS server is a security hole that can expose it to unwanted signals. (But see below.)
setfsgid will only succeed if the caller is the superuser or if fsgid matches either the real group ID, effective group ID, saved set-group-ID, or the current value of fsgid.
On success, the previous value of fsgid is returned. On error, the current value of fsgid is returned.
setfsgid is Linux specific and should not be used in programs intended to be portable. It is present since Linux 1.1.44 and in libc since libc 4.7.6.
No error messages of any kind are returned to the caller. At the very least, EPERM should be returned when the call fails.
When glibc determines that the argument is not a valid gid, it will return -1 and set errno to EINVAL without attempting the system call.
Note that at the time this system call was introduced, a process could send a signal to a process with the same effective user ID. Today signal permission handling is slightly different.