How to perform automated mining in regtest mode?

I'm trying to setup a private network of bitcoin nodes for testing and experimentation. From what i understand, i have two choices while setting up a network:

1. I can run bitcoind -testnet and connect to the public test network. The testnet has nodes that will facilitate the mining of my transactions as long as i just broadcast them.
2. I can configure the node with regtest=1 and manually run the generate command whenever i want to create blocks.

I've also managed to find out that the mining is no longer a part of the core codebase and is usually in an external component.

What i don't understand is how any of these mining softwares interact with the bitcoin node. Does it periodically poll through the JSON RPC to check if the transaction pool has reached a certain threshold (in numbers or size)? If not, what triggers the beginning of a mining operation? Can the bitcoin node send out events when the transaction pool reaches a threshold?

I want to setup a similar automated process for creating blocks in my regtest network, but using the generate command instead of any mining software. I don't need a robust PoW setup in my local network. My aim is to just build an app which interacts with the nodes and makes transactions. Is writing an RPC client which periodically queries the pool size and fires a generate command on reaching a threshold the right way to proceed? Or does the bitcoin node itself provide some way to setup periodic generation of blocks?

Thanks in advance.

P.S. I'm very new to bitcoin development, so apologies if this is an obvious question.