Often. And a firm that won't tell you when is a firm optimising for its own pipeline rather than your outcome.
Custom is wrong for commodity, regulated, standardised work. Payroll is the clearest example. The rules change constantly, the penalties for errors are severe, and ADP or Gusto have already solved it for a tiny fraction of what building it would cost. Same for the accounting ledger. QuickBooks and Xero are cheap, well understood by your accountant, and built to survive an audit. Build around them, connect to them, and leave the ledger alone.
Custom is wrong when the value is the network. If your clients already live inside a platform, collaborate with you there, and would have to change their own habits to work with you, replacing it has a migration cost that is social rather than technical. A custom client portal can absolutely be better than a shared platform seat, and it's still a conversation you have to have with every client. Count that.
Custom is wrong when you're too small for the arithmetic to work. If you're paying a few thousand dollars a year in total software spend, the platform is winning on price and that's the end of it. The maths tips when the licence bill is large, when it grows with headcount, and when you're working around the tool rather than with it.
Custom is wrong when nobody internally will own it. A system without an owner drifts. Requests pile up, nobody prioritises, and in two years it's as stale as the thing it replaced. You need one person with authority, time and interest. If you can't name them, wait.
Custom is wrong when your process genuinely changes every quarter in ways you can't describe. Not when it changes, because all processes change and a built system is easier to change than a configured one. When it changes faster than it can be specified. Settle it first.
And custom is wrong if you're six months from an acquisition, a merger or a major strategy shift. The acquirer will have opinions. Wait and see what they are.
The honest downsides of owning your system
You own the maintenance. There is no vendor rolling out improvements while you sleep. Security patches, dependency updates, browser changes and interface changes at connected systems all land on your desk. That's a standing cost and a standing responsibility.
You own the roadmap too, which sounds great until you realise it means nobody else is thinking about your software. A platform with ten thousand customers ships features you never asked for and sometimes love. Your system does exactly what you specified, and nothing more, until you decide otherwise and pay for it.
There's a key person consideration as well. The team that built it understands it best. Good documentation, clean conventions and standard technologies reduce that risk a lot, and they don't eliminate it. Ask what stack the system is built on, and if the answer isn't something a competent developer anywhere could pick up, ask why.
Finally, the first version will have rough edges. Every system does, bought or built. The difference is that with an owned system you can fix them, rather than filing a feature request into a queue you don't control.