From Brotato Wiki
This page was written as a friendly guide by the original author. It may not follow the same standards as the rest of the wiki, and may contain ideas and opinions that may not be considered objectively useful to every player or modder.
This guide was written by Darkly77, who made Invasion (among a few other bits).
Intro
Balance is a tricky thing to get right. From what I've seen, players want to feel like they're playing optimally, making good decisions instead of having items make the decisions for them.
This means that:
- If an item is too weak, players feel cheated.
- If an item is too strong, players feel like they're cheating for using it.
Seeing just 1 unbalanced item can change a player's opinion against not just one mod, but mods in general... so no pressure then!
Downsides
Downsides are important because they give the player interesting choices.
But an item doesn't have to have a downside. For items with specialised effects, their benefits are niche enough that, while they might help you, the real downside is the cost investment. This can be seen with certain vanilla items, eg:
- Turrets,
Piggy Bank, 
Scared Sausage, or 
Crown
So generally, a downside is needed if the positive is useful to the majority of builds, might not be if its effect is specialised.
Rarity
Common
Common items are probably the hardest to balance, because they appear so often, and set the overall direction your stats will learn towards.
A good example in vanilla is 
- Players often mentioned that HP Regen is weak, because its 3 commons all work against your other survivability -- reducing Life Steal, Consumable Healing, and Luck (ie. chance for consumable drops).
- While taking one of those common items won't have a huge impact on your run, having all 3 have downsides in the same area has a dramatic effect.
- The result being that HP Regen pushes you towards specialising in it very early, whereas other survivability buffs don't.
- This can have the effect of reducing player freedom, which can potentially make runs feel too similar and even boring -- so players often avoid HP Regen altogether.
Making decisions for the player takes away their freedom of choice. But by offering the player interesting and varied choices instead, items can be more engaging and fun.
Rare
Rare items can massively help turn the course of a run, taking you from behind the curve to just in front of it. They shouldn't be so OP that you're given a free win, but they don't need to be too specialised either, letting you offer strong stats that benefit most builds.
Legendary
If a Rare can tip the balance in your favor, a Legendary can be a run decider. No items should be completely OP, but it's OK for a Legendary item to come close to that, especially if it's specialised. For example:
Jet Pack is very strong, but because it's useful for pretty much any build, it's not so strong that it's an instant win.
Spider can potentially offer an enormous and run-deciding DPS increase, but it only reaches its full power potential when your build meets such less-used criteria.
Another thing to consider is that, while Rares are more likely to appear in shops, they're still very luck-dependant, whereas Legendaries can be guaranteed drops in higher Danger levels. This is why some players feel that higher Danger can actually be easier, because you can stock up on one or more Legendaries, and that can carry your run.
Underused Stats
If you feel like a certain stat is underused in vanilla, then it's probably underused for a reason -- eg. how Elemental Damage is often only useful to specific builds or loadouts. So if your modded items depend on one of these underused stats, one way to offer more chances to increase it is with items that have stacked positives.
This is where an item gives a slightly weaker version of 2 stats, one of them being something universally helpful (eg. Melee Damage), the other being your chosen overlooked stat. This lets players take it to invest in their general build, while essentially banking the second stat as a potential bonus for if/when your item that uses it appears, without making their build suffer for it, or bloating the item pool too much with potentially useless items.
Costs
There's a lot that could be said about costs, but it's probably easier to just refer to vanilla here. There's also a Balancing section on the Modding Notes page here on the wiki, which lists the average costs for items and weapons by tier:
Your own costs should fall somewhere within those average boundaries, depending on its power and versatility. Generally, powerful items cost more, while specialised ones are usually cheaper to pick up, giving the player a choice of taking something with potential utility later, or saving their cash for something more closely related to their current build.
Content Based on Other Media
I learned about this the hard way. When I made Invasion, I used items from the game Space Gladiators, including their effects. That game has 2 types of healing, so I just swapped them out for HP Regen and Life Steal. But this meant that HP Regen in particular became very prominent in Invasion, which made the game a bit too easy. So the tip here is: Add effects based on what makes sense for vanilla, rather than what's thematically appropriate for the thing your mod is based on.
Vanilla Balancing
The best tip I can offer is to check vanilla! Its items are exceptionally well-balanced. There's an interactive item grid that shows them here:
Other Modding Notes
Effects
The most interesting items are usually ones that add new effects to the game. But coding new effects can be tricky, especially if you're new to modding, so a good place to start is by looking at the effects from vanilla, and seeing if there are new ways you can use them. A list of vanilla's effects is available here:
I need to update that page for v0.8.0.3, but until I do, this mod migration guide covers what's change or removed, and in particular, lists the new effects from v0.8.
If you want to add your own effects, the code for BFX on GitHub is a good point of reference. You can search for one of its custom effect keys to see how it's implemented, which can teach you a lot about where some of the most important vanilla code is.