We recently had a customer who had a “bad experience”. Their medical billing and scheduling was completely shutdown for several days. They were the victim of an apparent hard drive failure. Their IT contractor had them set up to make daily backups onto a single hard drive. When their data was corrupted at some point they overwrote their only good backup.
What causes data corruptions? The four following factors are the primary causes of data corruptions:
1) Power failures or spikes
2) Improper shutdown
3) Network problems
4) Hardware failure
What can you do to minimize data corruptions?
Power Problems- Use a working Uninterruptible Power Supplies (UPS) on all computers. A UPS with a dead battery is just as likely to damage a database as using none at all.
Improper Shutdown- No matter what version of Microsoft Windows™ you are using it is imperative that you always shut your computer down properly. Turning the computer off without properly closing down Windows will eventually corrupt your Windows installation but more importantly will very likely damage any databases that are open.
To shut-down your computer properly, first shut down any programs that are open. Then click on the Start button and select the Shut Down option. You must then select the option to Shut Down and click OK.
Network Problems- Improper network Setup or design can definitely cause data corruption. Some network protocols are not fault tolerant. This means that they don’t handle problems well. Medisoft requires the use of the TCP/IP network protocol. Having other non-fault tolerant protocols running can cause data to travel across your network in ways that allow dropped packets of data to be lost. Likewise other network issues can cause problems.
Hardware failure- Prevention of hardware failures is expensive and not foolproof. There is a whole science to hardware fault tolerance. You can purchase RAID hard drive arrays which allow you to run multiple hard drives. You can purchase redundant power supplies, redundant network cards, you can even purchase redundant servers. While there is no question that these technologies can offer a higher level of protection they are not foolproof and your best protection in the long run is still incremental, high quality back ups.
Recovering from failures:More important, what can you do to recover from data corruptions? Your best option is to restore from your most recent uncorrupted backup. Once you have a problem it is too late, you must have a good backup routine in place prior to the failure.
Here is what you need to do:
1) Put a UPS on the server and all workstations- Use a working Uninterruptible Power Supply (UPS) on all computers. You must regularly check the batteries on the UPS on every computer.
2) Separate daily backups- You must make separate daily backups of your database. Do not overwrite your backup files. Create multiple backup files with different names and keep them somewhere safe.
3) Automatic backups- Medisoft software offers a capability to make regular scheduled backups. Use this feature! The Medisoft scheduled backup can be configured to make a separate backup each and every day.
4) Make sure all users exit properly from Medisoft- Make sure that you schedule your backup for a time that no one will be working in your system and make sure that every user properly exits from your system before the scheduled backup runs. If you have even one user in Medisoft when the automatic backup runs you will get an error message that some files were skipped and your backup will not be serviceable.
5) Test your backups- Test your backups periodically to verify the veracity of the backups. If the size of a backup file is 0 kb that means that you have no data in your backup. Back ups from one day to the next do not normally get smaller unless you run file maintenance or delete files or delete unnecessary files from your database folder (that could be very dangerous if you are not sure a file is unnecessary). Look at the size of your backups and make sure that they are the same or larger from one day to the next. If one day’s backup file is smaller than the one from the previous day without a reason then it is likely you have a bad backup file.
Need I say that HIPAA requires that you have a procedure in place not only to make regular backups but also that you test your backups at regular intervals?
6) Take backups off-site- Take a copy of your backups off site regularly. I have known customers that leave their one and only backup tape in the tape drive or laid on top of their server. What happens to their backup if the office burns down or somebody breaks in and steals the server?
How much data can you afford to reenter? The answer to that question should determine how frequently you should take your backups off-site.
7) Have a secondary backup capability- Why not make a redundant off site backup? High speed Internet is widely available now and backup capabilities have dropped in price to the point that they are cost effective. For less than $30 a month you can have a secondary backup stored in another state.
Several years ago I worked for a company that operated multiple servers for multiple functions when the Nimda virus struck. Nimda attacked HTML files where ever it could find them and actually propagated itself across a local network by traveling from computer to computer across network shares. It raged through our network like a grease fire. We were literally running around the office unplugging computers from the network (we actually kept many computers from being attacked by unplugging them). All the antivirus companies rushed to get updates out and some of the antivirus updates caused problems of their own. It took several hours to clean the entire network of all traces of the virus and then we restored the servers from backup tapes.
Here’s the interesting thing about some data failures, they propagate to your redundant hard drives or even redundant computers. For example, say that you run a mirrored server; anything bad that happens to one server automatically propagates to the other server! A virus problem like Nimda would simply infect your mirrored server or mirrored drives.
As they used to say in commercials years ago, “What is a mother to do?”
What is the best and ultimate solution? Make good incremental backups and store them off-site. Having one great backup is not enough; if you overwrite your one great backup with bad data it is no longer “great”.
Don’t have a “bad experience”. Make sure that your backup routines and procedures are adequate and redundant.
Mark Beans, MCSE, MCP+I
Medical Software Tools
http://www.medicalsoftwaretools.com/Labels: data corruptions, medical billing, medisoft, medisoft software, network problems, power failure
