If your goal is to modify the configuration and then repack it:
To open a Mikrotik backup file, you need to use a tool that can read the binary format of the file. There are a few options available: open mikrotik backup file repack
If you are determined to crack the format for research: If your goal is to modify the configuration
marcograss/routerosbackuptools: Tools to encrypt ... - GitHub open mikrotik backup file repack